[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:28 dist-test-slave-zmf4 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:28 dist-test-slave-zmf4 krb5kdc[399](info): set up 2 sockets
Jun 20 04:35:28 dist-test-slave-zmf4 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 04:35:29.193763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:29.324687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:29.438791 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:29.439163 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:29.439251 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:29.443394 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:29.443500 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:29.443536 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:29.443578 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:29.443610 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:29.449064 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:29.450245 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:29.451361 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:35:29.461012 412 server_base.cc:1034] running on GCE node
W20260620 04:35:29.461040 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.461047 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.461176 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:29.461843 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:29.462971 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:29.464246 412 hybrid_clock.cc:648] HybridClock initialized: now 1781930129464140 us; error 83 us; skew 500 ppm
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:29.472097 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:29.473242 412 webserver.cc:460] Webserver started at http://127.0.98.62:37003/ using document root <none> and password file <none>
I20260620 04:35:29.473817 412 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:29.473878 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:29.474054 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:29.476357 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "97b4b0ae4a0140e5baf46edb1c905601"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "bf3f2550e6fd3bcaa4e42c39d998ed36"
server_key_iv: "c0d8ee235bf5b732270cdc5c5415e45a"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.476929 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "97b4b0ae4a0140e5baf46edb1c905601"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "bf3f2550e6fd3bcaa4e42c39d998ed36"
server_key_iv: "c0d8ee235bf5b732270cdc5c5415e45a"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.480948 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:35:29.483557 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:29.485083 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:35:29.485234 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "97b4b0ae4a0140e5baf46edb1c905601"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "bf3f2550e6fd3bcaa4e42c39d998ed36"
server_key_iv: "c0d8ee235bf5b732270cdc5c5415e45a"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.485340 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:29.499074 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:29.502619 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:29.502795 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:29.510784 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39391
I20260620 04:35:29.510810 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39391 every 8 connection(s)
I20260620 04:35:29.511821 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:29.514840 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:29.520793 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: Bootstrap starting.
I20260620 04:35:29.521837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 412
I20260620 04:35:29.521960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:29.523252 392 external_mini_cluster.cc:1428] Setting key 95150f7accd711e08ece0613f3b2c71c
I20260620 04:35:29.523554 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:29.524395 480 log.cc:824] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:29.526504 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: No bootstrap required, opened a new log
I20260620 04:35:29.530642 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } }
I20260620 04:35:29.530893 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:29.530980 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b4b0ae4a0140e5baf46edb1c905601, State: Initialized, Role: FOLLOWER
I20260620 04:35:29.531430 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } }
I20260620 04:35:29.531581 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:29.531672 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:29.531792 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:29.532908 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } }
I20260620 04:35:29.533269 480 leader_election.cc:302] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b4b0ae4a0140e5baf46edb1c905601; no voters:
I20260620 04:35:29.533635 480 leader_election.cc:288] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:29.533941 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:29.534291 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [term 1 LEADER]: Becoming Leader. State: Replica: 97b4b0ae4a0140e5baf46edb1c905601, State: Running, Role: LEADER
I20260620 04:35:29.534704 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } }
I20260620 04:35:29.534984 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:29.536811 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b4b0ae4a0140e5baf46edb1c905601. Latest consensus state: current_term: 1 leader_uuid: "97b4b0ae4a0140e5baf46edb1c905601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } } }
I20260620 04:35:29.536931 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:29.536760 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b4b0ae4a0140e5baf46edb1c905601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b4b0ae4a0140e5baf46edb1c905601" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39391 } } }
I20260620 04:35:29.536984 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601 [sys.catalog]: This master's current role is: LEADER
W20260620 04:35:29.540618 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:35:29.540704 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:35:29.540764 497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:29.541610 497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:29.547840 497 catalog_manager.cc:1347] Generated new cluster ID: 2c67ff04c96e46eb8b1ae0431fbfa280
I20260620 04:35:29.547904 497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:29.551749 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:29.527281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.0.98.62:39391)
0620 04:35:29.527912 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:29.527933 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:29.529608 (+ 1675us) server_negotiation.cc:407] Connection header received
0620 04:35:29.532824 (+ 3216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:29.532845 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:29.533244 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:29.533604 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:29.537241 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.538823 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.540082 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.540652 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.540843 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:29.545275 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:29.545282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:29.545299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:29.545333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:29.548348 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:29.548915 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:29.548920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:29.548924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:29.549036 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:29.549392 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:29.549401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:29.549403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:29.549788 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:29.550011 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:29.550559 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 04:35:29.550828 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":141,"threads_started":1}
I20260620 04:35:29.568545 497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:29.569700 497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:29.575943 497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b4b0ae4a0140e5baf46edb1c905601: Generated new TSK 0
I20260620 04:35:29.576819 497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:29.613413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:29.724170 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:29.724509 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:29.724593 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:29.728955 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:29.729045 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:29.729139 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:29.734460 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:29.735618 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:29.736784 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:29.746520 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.746565 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.746536 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:29.746783 506 server_base.cc:1034] running on GCE node
I20260620 04:35:29.747228 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:29.747870 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:29.749045 506 hybrid_clock.cc:648] HybridClock initialized: now 1781930129749009 us; error 48 us; skew 500 ppm
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:29.752003 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:29.753067 506 webserver.cc:460] Webserver started at http://127.0.98.1:37753/ using document root <none> and password file <none>
I20260620 04:35:29.753700 506 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:29.753801 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:29.754029 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:29.756188 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee60b3ebc47c4c51995c4b5912f5f534"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "7afd5869d315df211511643891ac7203"
server_key_iv: "718933ae3728374c4f291b37b24eed25"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.756872 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee60b3ebc47c4c51995c4b5912f5f534"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "7afd5869d315df211511643891ac7203"
server_key_iv: "718933ae3728374c4f291b37b24eed25"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.761049 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:35:29.763695 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:29.765017 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:35:29.765120 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "ee60b3ebc47c4c51995c4b5912f5f534"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "7afd5869d315df211511643891ac7203"
server_key_iv: "718933ae3728374c4f291b37b24eed25"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.765205 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:29.797626 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:29.801263 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:29.801421 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:29.801964 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:29.802762 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:29.802809 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:29.802848 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:29.802865 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:29.812850 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39217
I20260620 04:35:29.812870 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39217 every 8 connection(s)
I20260620 04:35:29.813733 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:29.820478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 506
I20260620 04:35:29.820586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:29.820840 392 external_mini_cluster.cc:1428] Setting key 50d77243f93ff50b3f3b4e12bb865829
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:29.829600 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:29.815576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41423 (local address 127.0.98.62:39391)
0620 04:35:29.815782 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:29.815785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:29.816677 (+ 892us) server_negotiation.cc:407] Connection header received
0620 04:35:29.818582 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:29.818587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:29.818670 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:29.818759 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:29.820443 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.821215 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.822577 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.822935 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.823061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:29.825630 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:29.825633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:29.825635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:29.825657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:29.827285 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:29.827830 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:29.827841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:29.827842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:29.827917 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:29.828458 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:29.828461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:29.828463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:29.828676 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:29.828793 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:29.829300 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 04:35:29.829432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:29.830474 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:29.815938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39391 (local address 127.0.98.1:41423)
0620 04:35:29.816509 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:29.816563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:29.818296 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:29.819097 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:29.819107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:29.819577 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:29.820269 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:29.820293 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.821374 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.821378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:29.822450 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:29.822459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:29.823105 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:29.823109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:29.823257 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:29.823923 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:29.823950 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:29.825451 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:29.827422 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:29.827428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:29.827443 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:29.827716 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:29.828019 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:29.828022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:29.828024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:29.828341 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:29.828791 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:29.828798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:29.829050 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 04:35:29.829244 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 04:35:29.829587 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":123,"threads_started":1}
I20260620 04:35:29.831977 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39391
I20260620 04:35:29.832257 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:35:29.832891 635 heartbeater.cc:511] Master 127.0.98.62:39391 requested a full tablet report, sending...
I20260620 04:35:29.834547 444 ts_manager.cc:194] Registered new tserver with Master: ee60b3ebc47c4c51995c4b5912f5f534 (127.0.98.1:39217)
I20260620 04:35:29.835793 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:29.851557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:29.965065 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:29.965356 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:29.965399 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:29.969656 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:29.969708 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:29.969765 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:29.974784 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:29.975703 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:29.976831 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:35:29.986724 640 server_base.cc:1034] running on GCE node
W20260620 04:35:29.986747 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.986788 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:29.986750 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:29.987356 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:29.988015 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:29.989248 640 hybrid_clock.cc:648] HybridClock initialized: now 1781930129989211 us; error 47 us; skew 500 ppm
Jun 20 04:35:29 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:29.992144 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:29.993093 640 webserver.cc:460] Webserver started at http://127.0.98.2:46691/ using document root <none> and password file <none>
I20260620 04:35:29.993651 640 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:29.993741 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:29.993963 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:29.996083 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e209cc06d9348da8a384d666d4579e6"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "01aa4b20b5f75fa992d8c27f04266fbb"
server_key_iv: "46386405dfebbef9c1c47bb9683be0f9"
server_key_version: "encryptionkey@0"
I20260620 04:35:29.996729 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e209cc06d9348da8a384d666d4579e6"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "01aa4b20b5f75fa992d8c27f04266fbb"
server_key_iv: "46386405dfebbef9c1c47bb9683be0f9"
server_key_version: "encryptionkey@0"
I20260620 04:35:30.000604 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:35:30.003120 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.004608 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:30.004766 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "1e209cc06d9348da8a384d666d4579e6"
format_stamp: "Formatted at 2026-06-20 04:35:29 on dist-test-slave-zmf4"
server_key: "01aa4b20b5f75fa992d8c27f04266fbb"
server_key_iv: "46386405dfebbef9c1c47bb9683be0f9"
server_key_version: "encryptionkey@0"
I20260620 04:35:30.004890 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:30.021170 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:30.024433 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:30.024664 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:30.025298 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:30.026162 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:30.026242 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.026383 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:30.026471 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.037398 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41919
I20260620 04:35:30.037439 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41919 every 8 connection(s)
I20260620 04:35:30.038446 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:30.048604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 640
I20260620 04:35:30.048756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:30.049043 392 external_mini_cluster.cc:1428] Setting key 2b80610a9fdd7583b8f2e8552e0c4591
I20260620 04:35:30.053125 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.040430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60989 (local address 127.0.98.62:39391)
0620 04:35:30.040561 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:30.040565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:30.041500 (+ 935us) server_negotiation.cc:407] Connection header received
0620 04:35:30.042823 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:30.042827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:30.042881 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:30.042962 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:30.044070 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.044666 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.045791 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.046021 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.046132 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.048757 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:30.048760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:30.048762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:30.048786 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:30.050510 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.051102 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.051113 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.051115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.051194 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.051577 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.051581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.051583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.051920 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:30.052039 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:30.052751 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 04:35:30.052905 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:30.054113 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.040852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39391 (local address 127.0.98.2:60989)
0620 04:35:30.041350 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:30.041392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:30.042618 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:30.043122 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:30.043129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:30.043481 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:30.043932 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:30.043944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.044799 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.044807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:30.045667 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:30.045675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.046165 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.046169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:30.046338 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.046905 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:30.046922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:30.048588 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:30.050655 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:30.050661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:30.050678 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:30.050968 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:30.051300 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:30.051304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:30.051306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:30.051461 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:30.052062 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:30.052071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:30.052449 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 04:35:30.052681 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 04:35:30.053128 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260620 04:35:30.055357 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39391
I20260620 04:35:30.055589 769 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:30.056052 769 heartbeater.cc:511] Master 127.0.98.62:39391 requested a full tablet report, sending...
I20260620 04:35:30.057263 444 ts_manager.cc:194] Registered new tserver with Master: 1e209cc06d9348da8a384d666d4579e6 (127.0.98.2:41919)
I20260620 04:35:30.058073 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60989
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:30.079319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:30.194633 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:30.194886 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:30.194931 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:30.198968 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:30.199026 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:30.199085 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:30.204838 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39391
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:30.205854 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:30.206811 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:30.217098 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:30.217099 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:30.217518 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:30.217685 774 server_base.cc:1034] running on GCE node
I20260620 04:35:30.218042 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:30.218652 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:30.219841 774 hybrid_clock.cc:648] HybridClock initialized: now 1781930130219803 us; error 48 us; skew 500 ppm
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:30.222661 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:30.223562 774 webserver.cc:460] Webserver started at http://127.0.98.3:36811/ using document root <none> and password file <none>
I20260620 04:35:30.224119 774 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:30.224227 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:30.224459 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:30.226410 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "9909503bcd3d41c0953b5600e921dfd3"
format_stamp: "Formatted at 2026-06-20 04:35:30 on dist-test-slave-zmf4"
server_key: "8ab4cbd81c4b6d1eca984059840bd0ab"
server_key_iv: "0c6c659871603d12aa4a18185a4e7c44"
server_key_version: "encryptionkey@0"
I20260620 04:35:30.226989 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9909503bcd3d41c0953b5600e921dfd3"
format_stamp: "Formatted at 2026-06-20 04:35:30 on dist-test-slave-zmf4"
server_key: "8ab4cbd81c4b6d1eca984059840bd0ab"
server_key_iv: "0c6c659871603d12aa4a18185a4e7c44"
server_key_version: "encryptionkey@0"
I20260620 04:35:30.230772 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 04:35:30.233204 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.234463 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:30.234690 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "9909503bcd3d41c0953b5600e921dfd3"
format_stamp: "Formatted at 2026-06-20 04:35:30 on dist-test-slave-zmf4"
server_key: "8ab4cbd81c4b6d1eca984059840bd0ab"
server_key_iv: "0c6c659871603d12aa4a18185a4e7c44"
server_key_version: "encryptionkey@0"
I20260620 04:35:30.234807 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:30.254212 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:30.257236 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:30.257463 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:30.258018 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:30.258809 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:30.258878 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.258949 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:30.259002 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:30.269073 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38879
I20260620 04:35:30.269232 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38879 every 8 connection(s)
I20260620 04:35:30.270067 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:30.275990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 774
I20260620 04:35:30.276103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:30.276417 392 external_mini_cluster.cc:1428] Setting key a09ee1f236614734e0b26a73ae21fa81
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:30.283906 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.271905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58245 (local address 127.0.98.62:39391)
0620 04:35:30.272091 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:30.272095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:30.272945 (+ 850us) server_negotiation.cc:407] Connection header received
0620 04:35:30.274256 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:30.274260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:30.274316 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:30.274421 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:30.275524 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.276279 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.277638 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.277896 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.278011 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.280488 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:30.280493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:30.280502 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:30.280524 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:30.281892 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.282444 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.282449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.282450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.282530 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.282893 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.282896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.282898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.283072 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:30.283179 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:30.283641 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 04:35:30.283752 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 04:35:30.284826 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.272315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39391 (local address 127.0.98.3:58245)
0620 04:35:30.272796 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:30.272837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:30.274050 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:30.274581 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:30.274587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:30.274928 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:30.275387 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:30.275398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.276404 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.276408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:30.277532 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:30.277542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.278047 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.278051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:30.278226 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.278815 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:30.278857 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:30.280338 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:30.282056 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:30.282062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:30.282076 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:30.282336 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:30.282630 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:30.282633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:30.282635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:30.282781 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:30.283174 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:30.283180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:30.283415 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 04:35:30.283601 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 04:35:30.283913 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260620 04:35:30.286000 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39391
I20260620 04:35:30.286207 903 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:30.286624 903 heartbeater.cc:511] Master 127.0.98.62:39391 requested a full tablet report, sending...
I20260620 04:35:30.287583 444 ts_manager.cc:194] Registered new tserver with Master: 9909503bcd3d41c0953b5600e921dfd3 (127.0.98.3:38879)
I20260620 04:35:30.288142 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58245
I20260620 04:35:30.290683 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:30.316030 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.306303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.98.1:39217)
0620 04:35:30.306637 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:30.306650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:30.306674 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:35:30.306775 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:30.306781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:30.306959 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:30.307072 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:30.307755 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.308794 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.309676 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.310135 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.310239 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.312626 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:30.312633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:30.312646 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:30.312669 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:30.314322 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.314766 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.314772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.314774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.314865 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.315091 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.315093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.315094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.315314 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:30.315471 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:30.315681 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 04:35:30.315846 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":137,"threads_started":1}
W20260620 04:35:30.317000 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48868
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:30.343367 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:30.334541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.0.98.1:39217)
0620 04:35:30.334737 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:30.334741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:30.334871 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:35:30.334974 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:30.334976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:30.335014 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:30.335079 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:30.335829 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.336391 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.337248 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:30.337477 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:30.337559 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:30.340096 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:30.340106 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:30.340108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:30.340129 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:30.341605 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.342067 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.342070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.342071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.342165 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:30.342508 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:30.342511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:30.342513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:30.342682 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:30.342796 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:30.343035 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:35:30.343132 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 04:35:30.344336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 506
I20260620 04:35:30.350593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 640
I20260620 04:35:30.357856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 774
I20260620 04:35:30.362991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 412
2026-06-20T04:35:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2240 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[917](info): set up 2 sockets
Jun 20 04:35:30 dist-test-slave-zmf4 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 04:35:31.428124 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:31 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:31.556710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:31.665391 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:31.665720 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:31.665794 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:31.670087 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:31.670181 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:31.670218 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:31.670277 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:31.670339 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:31.675707 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:31.676780 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:31.677805 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:31.686015 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:31.686105 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:31.686030 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:31.686352 930 server_base.cc:1034] running on GCE node
I20260620 04:35:31.686934 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:31.687994 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:31.689200 930 hybrid_clock.cc:648] HybridClock initialized: now 1781930131689154 us; error 62 us; skew 500 ppm
Jun 20 04:35:31 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:31.692510 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:31.693501 930 webserver.cc:460] Webserver started at http://127.0.98.62:42561/ using document root <none> and password file <none>
I20260620 04:35:31.694123 930 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:31.694228 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:31.694460 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:31.696715 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "280aa326685f4f71b9a8ce33ffdb237d"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "dd1cef88da1e1ffff5bf499fdda0348b"
server_key_iv: "d3c2473ed9617d76aa9dc8032fdd737b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.697329 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "280aa326685f4f71b9a8ce33ffdb237d"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "dd1cef88da1e1ffff5bf499fdda0348b"
server_key_iv: "d3c2473ed9617d76aa9dc8032fdd737b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.701326 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:35:31.703850 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:31.705230 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:31.705384 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "280aa326685f4f71b9a8ce33ffdb237d"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "dd1cef88da1e1ffff5bf499fdda0348b"
server_key_iv: "d3c2473ed9617d76aa9dc8032fdd737b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.705494 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:31.752364 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:31.755777 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:31.756001 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:31.763545 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35411
I20260620 04:35:31.763563 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35411 every 8 connection(s)
I20260620 04:35:31.764600 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:31.767256 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:31.772758 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d: Bootstrap starting.
I20260620 04:35:31.773463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 930
I20260620 04:35:31.773582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:31.773947 392 external_mini_cluster.cc:1428] Setting key f736c5a2f03435d5df9563b5f78a1ea1
I20260620 04:35:31.775862 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:31.776666 998 log.cc:824] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:31.778463 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d: No bootstrap required, opened a new log
I20260620 04:35:31.782042 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } }
I20260620 04:35:31.782339 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:31.782392 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280aa326685f4f71b9a8ce33ffdb237d, State: Initialized, Role: FOLLOWER
Jun 20 04:35:31 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:31.782879 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } }
I20260620 04:35:31.783041 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:31.783130 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:31.783233 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:31.784394 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } }
I20260620 04:35:31.784751 998 leader_election.cc:302] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280aa326685f4f71b9a8ce33ffdb237d; no voters:
I20260620 04:35:31.785131 998 leader_election.cc:288] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:31.785202 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:31.785405 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [term 1 LEADER]: Becoming Leader. State: Replica: 280aa326685f4f71b9a8ce33ffdb237d, State: Running, Role: LEADER
I20260620 04:35:31.785703 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } }
I20260620 04:35:31.786412 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:31.786818 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280aa326685f4f71b9a8ce33ffdb237d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } } }
I20260620 04:35:31.786936 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:31.787277 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280aa326685f4f71b9a8ce33ffdb237d. Latest consensus state: current_term: 1 leader_uuid: "280aa326685f4f71b9a8ce33ffdb237d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280aa326685f4f71b9a8ce33ffdb237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35411 } } }
I20260620 04:35:31.787397 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:31.788185 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:31.788986 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:31.775349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.62:35411)
0620 04:35:31.775788 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:31.775802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:31.775839 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:35:31.776571 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:31.776590 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:31.776966 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:31.777314 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:31.778070 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:31.779102 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:31.779953 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:31.780493 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:31.780669 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:31.783461 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:31.783466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:31.783479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:31.783505 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:31.785590 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:31.786099 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:31.786106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:31.786112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:31.786241 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:31.786568 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:31.786572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:31.786573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:31.787224 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:31.787509 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:31.787659 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 04:35:31.787978 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":126,"threads_started":1}
I20260620 04:35:31.791955 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:31.797783 1007 catalog_manager.cc:1347] Generated new cluster ID: 82a7009ffb7447fb94c366dbe0a3f20e
I20260620 04:35:31.797848 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:31.810396 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:31.811314 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:31.816551 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280aa326685f4f71b9a8ce33ffdb237d: Generated new TSK 0
I20260620 04:35:31.817242 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:31.848425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:31.959631 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:31.960044 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:31.960137 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:31.964303 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:31.964382 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:31.964476 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:31.969784 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:31.970778 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:31.971743 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:31.981010 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:31.981027 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:31.981070 1024 server_base.cc:1034] running on GCE node
W20260620 04:35:31.981063 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:31.981712 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:31.982307 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:31.983503 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781930131983470 us; error 46 us; skew 500 ppm
Jun 20 04:35:31 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:31.986500 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:31.987521 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39769/ using document root <none> and password file <none>
I20260620 04:35:31.988076 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:31.988143 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:31.988489 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:31.990705 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "573df7c27837422f94e310a6663f29c1"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "89c0f9a7199ff08563b1123a065a3dd6"
server_key_iv: "daeb9fa826187aff156742f21f27137b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.991250 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "573df7c27837422f94e310a6663f29c1"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "89c0f9a7199ff08563b1123a065a3dd6"
server_key_iv: "daeb9fa826187aff156742f21f27137b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.995563 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:35:31.997912 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:31.999157 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:35:31.999255 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "573df7c27837422f94e310a6663f29c1"
format_stamp: "Formatted at 2026-06-20 04:35:31 on dist-test-slave-zmf4"
server_key: "89c0f9a7199ff08563b1123a065a3dd6"
server_key_iv: "daeb9fa826187aff156742f21f27137b"
server_key_version: "encryptionkey@0"
I20260620 04:35:31.999401 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:32.029299 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:32.032835 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:32.033059 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:32.033793 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:32.034622 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:32.034701 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.034772 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:32.034808 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.043448 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34877
I20260620 04:35:32.043469 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34877 every 8 connection(s)
I20260620 04:35:32.044348 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:32.054579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1024
I20260620 04:35:32.054704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:32.055001 392 external_mini_cluster.cc:1428] Setting key a3ead38d33b5daaf499b38102c7017fc
I20260620 04:35:32.058527 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.046235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58667 (local address 127.0.98.62:35411)
0620 04:35:32.046385 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.046390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.047286 (+ 896us) server_negotiation.cc:407] Connection header received
0620 04:35:32.048632 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.048636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.048679 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.048759 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.049854 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.050383 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.051526 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.051768 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.051878 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.054734 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.054739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.054745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.054776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.056415 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.056883 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.056890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.056895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.056982 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.057308 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.057314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.057319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.057595 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.057730 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.058233 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.058361 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:35:32.059310 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.046659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35411 (local address 127.0.98.1:58667)
0620 04:35:32.047111 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.047163 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.048411 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.048915 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.048921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.049263 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:32.049698 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.049711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.050518 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.050521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.051396 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.051406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.051962 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.051966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.052128 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.052776 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:32.052793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:32.054531 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:32.056514 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.056519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.056530 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.056779 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.057085 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.057087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.057089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.057210 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.057726 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:32.057731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:32.057958 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.058191 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.058492 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260620 04:35:32.060647 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35411
I20260620 04:35:32.060916 1153 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:32.061405 1153 heartbeater.cc:511] Master 127.0.98.62:35411 requested a full tablet report, sending...
I20260620 04:35:32.062904 962 ts_manager.cc:194] Registered new tserver with Master: 573df7c27837422f94e310a6663f29c1 (127.0.98.1:34877)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:32.064152 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:32.083956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:32.193310 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:32.193552 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:32.193593 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:32.197420 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:32.197468 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:32.197525 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:32.202611 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:32.203560 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:32.204481 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:32.213860 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:32.213860 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:32.213929 1158 server_base.cc:1034] running on GCE node
W20260620 04:35:32.213855 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:32.214411 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:32.215040 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:32.216279 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781930132216245 us; error 45 us; skew 500 ppm
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:32.219354 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:32.220278 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35229/ using document root <none> and password file <none>
I20260620 04:35:32.220819 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:32.220880 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:32.221091 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:32.223009 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd62d59a91994b94aeb355c5be11f66a"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "fefa30456cfb08282c1900eacebdc6cc"
server_key_iv: "535d29e6ddf361d743c8620093c891a3"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.223557 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd62d59a91994b94aeb355c5be11f66a"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "fefa30456cfb08282c1900eacebdc6cc"
server_key_iv: "535d29e6ddf361d743c8620093c891a3"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.227082 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:35:32.229446 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.230643 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:32.230795 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "cd62d59a91994b94aeb355c5be11f66a"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "fefa30456cfb08282c1900eacebdc6cc"
server_key_iv: "535d29e6ddf361d743c8620093c891a3"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.230906 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:32.270066 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:32.273336 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:32.273566 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:32.274245 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:32.275090 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:32.275324 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.275441 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:32.275477 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.284855 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34899
I20260620 04:35:32.284907 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34899 every 8 connection(s)
I20260620 04:35:32.285959 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:32.290360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1158
I20260620 04:35:32.290455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:32.290654 392 external_mini_cluster.cc:1428] Setting key d4d01a6f46d1220206332ac0e497ece6
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:32.304109 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.288331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47447 (local address 127.0.98.62:35411)
0620 04:35:32.288491 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.288496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.289857 (+ 1361us) server_negotiation.cc:407] Connection header received
0620 04:35:32.292878 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.292881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.292926 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.293020 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.295288 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.295814 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.297069 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.297302 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.297414 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.299970 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.299973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.299975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.299996 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.301791 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.302383 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.302388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.302390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.302469 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.302867 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.302869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.302871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.303033 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.303110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.303829 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.303925 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:35:32.305423 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.288788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35411 (local address 127.0.98.2:47447)
0620 04:35:32.289689 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.289737 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.292501 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.293351 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.293366 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.294069 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:32.295111 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.295136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.295948 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.295952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.296944 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.296953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.297458 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.297462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.297639 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.298313 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:32.298329 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:32.299795 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:32.301918 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.301925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.301941 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.302270 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.302603 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.302606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.302607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.302774 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.303153 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:32.303159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:32.303461 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.303765 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.304185 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260620 04:35:32.306655 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35411
I20260620 04:35:32.306921 1287 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:32.307439 1287 heartbeater.cc:511] Master 127.0.98.62:35411 requested a full tablet report, sending...
I20260620 04:35:32.308918 962 ts_manager.cc:194] Registered new tserver with Master: cd62d59a91994b94aeb355c5be11f66a (127.0.98.2:34899)
I20260620 04:35:32.309582 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:32.323860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:32.439059 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:32.439356 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:32.439433 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:32.443454 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:32.443538 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:32.443632 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:32.448845 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:32.449872 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:32.450891 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:35:32.460613 1292 server_base.cc:1034] running on GCE node
W20260620 04:35:32.460821 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:32.460819 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:32.460888 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:32.461385 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:32.462003 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:32.463207 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781930132463191 us; error 32 us; skew 500 ppm
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:32.466111 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:32.467059 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39445/ using document root <none> and password file <none>
I20260620 04:35:32.467800 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:32.467891 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:32.468142 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:32.471107 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6b397d8f390492b962d280bda05b9d4"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "e944ad6272dbb9eea57ee716a296dc2a"
server_key_iv: "a677ac5c4e34af064401785f1a62fcd5"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.471833 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6b397d8f390492b962d280bda05b9d4"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "e944ad6272dbb9eea57ee716a296dc2a"
server_key_iv: "a677ac5c4e34af064401785f1a62fcd5"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.476609 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 04:35:32.479383 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.480793 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:32.480942 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "b6b397d8f390492b962d280bda05b9d4"
format_stamp: "Formatted at 2026-06-20 04:35:32 on dist-test-slave-zmf4"
server_key: "e944ad6272dbb9eea57ee716a296dc2a"
server_key_iv: "a677ac5c4e34af064401785f1a62fcd5"
server_key_version: "encryptionkey@0"
I20260620 04:35:32.481065 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:32.500530 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:32.503944 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:32.504156 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:32.504779 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:32.505578 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:32.505631 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.505669 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:32.505684 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:32.515159 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44449
I20260620 04:35:32.515182 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44449 every 8 connection(s)
I20260620 04:35:32.516048 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:32.520514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1292
I20260620 04:35:32.520629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:32.520920 392 external_mini_cluster.cc:1428] Setting key c36e874858f193c48f54cd3c88bcf600
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:32.530701 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.517903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46567 (local address 127.0.98.62:35411)
0620 04:35:32.518046 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.518049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.518891 (+ 842us) server_negotiation.cc:407] Connection header received
0620 04:35:32.520335 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.520339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.520405 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.520509 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.522095 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.522729 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.523963 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.524286 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.524394 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.526973 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.526976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.526979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.527004 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.528460 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.528982 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.528985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.528986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.529067 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.529487 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.529490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.529491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.529790 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.529880 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.530397 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.530580 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 04:35:32.531400 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.518276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35411 (local address 127.0.98.3:46567)
0620 04:35:32.518739 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.518783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.520065 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.520754 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.520761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.521188 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:32.521940 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.521965 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.522864 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.522868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.523836 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.523845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.524440 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.524443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.524590 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.525209 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:32.525224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:32.526741 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:32.528578 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.528583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.528594 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.528860 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.529202 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:32.529205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:32.529206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:32.529350 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:32.529925 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:32.529929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:32.530145 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.530319 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.530556 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260620 04:35:32.532697 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35411
I20260620 04:35:32.532944 1421 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:32.533707 1421 heartbeater.cc:511] Master 127.0.98.62:35411 requested a full tablet report, sending...
I20260620 04:35:32.534960 961 ts_manager.cc:194] Registered new tserver with Master: b6b397d8f390492b962d280bda05b9d4 (127.0.98.3:44449)
I20260620 04:35:32.535529 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46567
I20260620 04:35:32.546829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:35:32.557716 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.550225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.0.98.62:35411)
0620 04:35:32.550418 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.550422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.550552 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:35:32.550651 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.550653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.550704 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.550780 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.551529 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.552038 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.552962 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.553176 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.553264 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.554671 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.554674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.554676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.554697 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.556052 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.556503 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.556506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.556507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.556589 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.556882 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.556884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.556885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.557109 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.557231 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.557499 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.557601 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 04:35:32.562481 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:32.565420 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:32.580264 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.573926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44449 (local address 127.0.0.1:57870)
0620 04:35:32.574296 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.574347 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.574470 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.575448 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.575459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.575489 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:32.575643 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.575651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.577200 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.577206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.578632 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.578646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.579671 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.579674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.579798 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.579841 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.579958 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.580072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260620 04:35:32.580544 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.574093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.0.98.3:44449)
0620 04:35:32.574912 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.574924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.574950 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:35:32.575031 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.575038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.575253 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.575825 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:32.575997 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.577049 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.578784 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.579548 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.579665 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.579811 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.580007 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.580126 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":135,"threads_started":1}
I20260620 04:35:32.580607 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.574151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34899 (local address 127.0.0.1:34046)
0620 04:35:32.574728 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.574742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.574976 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.575738 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.575742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.575759 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:32.575833 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.575838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.577642 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.577646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.579009 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.579020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.580177 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.580181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.580303 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.580327 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.580386 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.580454 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":67,"thread_start_us":104,"threads_started":1}
I20260620 04:35:32.581010 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.575360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34877 (local address 127.0.0.1:44328)
0620 04:35:32.575848 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.575862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.575963 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.576446 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.576449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.576469 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:32.576554 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.576560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.578016 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.578019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.579447 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.579455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.580609 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.580613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.580702 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.580723 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.580806 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.580879 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260620 04:35:32.581350 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.575065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.0.98.1:34877)
0620 04:35:32.575717 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.575724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.575973 (+ 249us) server_negotiation.cc:407] Connection header received
0620 04:35:32.576066 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.576072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.576304 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.576455 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:32.576682 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.577889 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.579566 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.580475 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.580600 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.580750 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.580835 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.580920 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":81,"threads_started":1}
I20260620 04:35:32.582024 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.574743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.0.98.2:34899)
0620 04:35:32.575237 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.575250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.575273 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:35:32.575350 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.575356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.575552 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.575991 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:32.576141 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.577484 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.579147 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.580034 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.581587 (+ 1553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.581738 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.581790 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.581876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":137,"threads_started":1}
I20260620 04:35:32.583696 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0a77b2965333440f8590fb447e8d3358 (DEFAULT_TABLE table=test-table [id=fd28fac9375449eda46aaaf4e78a5d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:32.584803 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a77b2965333440f8590fb447e8d3358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:32.584556 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0a77b2965333440f8590fb447e8d3358 (DEFAULT_TABLE table=test-table [id=fd28fac9375449eda46aaaf4e78a5d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:32.584558 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0a77b2965333440f8590fb447e8d3358 (DEFAULT_TABLE table=test-table [id=fd28fac9375449eda46aaaf4e78a5d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:32.585510 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a77b2965333440f8590fb447e8d3358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:32.585510 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a77b2965333440f8590fb447e8d3358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:32.592407 1438 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Bootstrap starting.
I20260620 04:35:32.593642 1440 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: Bootstrap starting.
I20260620 04:35:32.594679 1438 tablet_bootstrap.cc:654] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:32.595314 1440 tablet_bootstrap.cc:654] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:32.595898 1438 log.cc:824] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:32.596119 1440 log.cc:824] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:32.596621 1439 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: Bootstrap starting.
I20260620 04:35:32.597842 1440 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: No bootstrap required, opened a new log
I20260620 04:35:32.598024 1440 ts_tablet_manager.cc:1397] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:35:32.598291 1439 tablet_bootstrap.cc:654] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:32.599236 1439 log.cc:824] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:32.601068 1438 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: No bootstrap required, opened a new log
I20260620 04:35:32.601159 1439 tablet_bootstrap.cc:492] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: No bootstrap required, opened a new log
I20260620 04:35:32.601289 1438 ts_tablet_manager.cc:1397] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260620 04:35:32.601332 1439 ts_tablet_manager.cc:1397] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:35:32.601873 1440 raft_consensus.cc:348] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.602126 1440 raft_consensus.cc:374] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:32.602183 1440 raft_consensus.cc:729] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573df7c27837422f94e310a6663f29c1, State: Initialized, Role: FOLLOWER
I20260620 04:35:32.603040 1440 consensus_queue.cc:260] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.603999 1440 ts_tablet_manager.cc:1428] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 04:35:32.604372 1153 heartbeater.cc:503] Master 127.0.98.62:35411 was elected leader, sending a full tablet report...
I20260620 04:35:32.605224 1439 raft_consensus.cc:348] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.605507 1439 raft_consensus.cc:374] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:32.605578 1439 raft_consensus.cc:729] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd62d59a91994b94aeb355c5be11f66a, State: Initialized, Role: FOLLOWER
I20260620 04:35:32.605809 1438 raft_consensus.cc:348] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.605988 1439 consensus_queue.cc:260] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.606055 1438 raft_consensus.cc:374] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:32.606124 1438 raft_consensus.cc:729] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b397d8f390492b962d280bda05b9d4, State: Initialized, Role: FOLLOWER
I20260620 04:35:32.606629 1287 heartbeater.cc:503] Master 127.0.98.62:35411 was elected leader, sending a full tablet report...
I20260620 04:35:32.606547 1438 consensus_queue.cc:260] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.606913 1439 ts_tablet_manager.cc:1428] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:35:32.607425 1438 ts_tablet_manager.cc:1428] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260620 04:35:32.607771 1421 heartbeater.cc:503] Master 127.0.98.62:35411 was elected leader, sending a full tablet report...
I20260620 04:35:32.664619 1445 raft_consensus.cc:482] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:32.664856 1445 raft_consensus.cc:504] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.666035 1445 leader_election.cc:288] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b397d8f390492b962d280bda05b9d4 (127.0.98.3:44449), 573df7c27837422f94e310a6663f29c1 (127.0.98.1:34877)
I20260620 04:35:32.670379 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.666394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39547 (local address 127.0.98.3:44449)
0620 04:35:32.666640 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.666644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.667140 (+ 496us) server_negotiation.cc:407] Connection header received
0620 04:35:32.667236 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.667239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.667296 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.667374 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:32.667632 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.668291 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.669401 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.669838 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.669950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.670021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.670198 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.670268 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 04:35:32.670506 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.666312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34877 (local address 127.0.98.2:39067)
0620 04:35:32.666435 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.666451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.666626 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.667071 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.667074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.667122 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:32.667258 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.667265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.668227 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.668231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.669150 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.669159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.670096 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.670099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.670208 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.670229 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.670300 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.670364 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":45}
I20260620 04:35:32.670396 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.666321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44449 (local address 127.0.98.2:39547)
0620 04:35:32.667030 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:32.667044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:32.667138 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:32.667402 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:32.667405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:32.667419 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:32.667521 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.667526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.668428 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.668431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.669277 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:32.669287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.669964 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.669969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:32.670076 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.670097 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:35:32.670162 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:35:32.670246 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":334,"thread_start_us":129,"threads_started":1}
I20260620 04:35:32.670656 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.666637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39067 (local address 127.0.98.1:34877)
0620 04:35:32.666790 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.666794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.666808 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:35:32.666877 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.666881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.666932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.667021 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:32.667418 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.667992 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.669286 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.669881 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.670048 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.670108 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.670451 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.670529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 04:35:32.671223 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a77b2965333440f8590fb447e8d3358" candidate_uuid: "cd62d59a91994b94aeb355c5be11f66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573df7c27837422f94e310a6663f29c1" is_pre_election: true
I20260620 04:35:32.671443 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a77b2965333440f8590fb447e8d3358" candidate_uuid: "cd62d59a91994b94aeb355c5be11f66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b397d8f390492b962d280bda05b9d4" is_pre_election: true
I20260620 04:35:32.671526 1107 raft_consensus.cc:2454] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd62d59a91994b94aeb355c5be11f66a in term 0.
I20260620 04:35:32.671715 1375 raft_consensus.cc:2454] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd62d59a91994b94aeb355c5be11f66a in term 0.
I20260620 04:35:32.672026 1176 leader_election.cc:302] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573df7c27837422f94e310a6663f29c1, cd62d59a91994b94aeb355c5be11f66a; no voters:
I20260620 04:35:32.672329 1445 raft_consensus.cc:2781] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:32.672415 1445 raft_consensus.cc:482] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:32.672459 1445 raft_consensus.cc:3037] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:32.673470 1445 raft_consensus.cc:504] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.673800 1445 leader_election.cc:288] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [CANDIDATE]: Term 1 election: Requested vote from peers b6b397d8f390492b962d280bda05b9d4 (127.0.98.3:44449), 573df7c27837422f94e310a6663f29c1 (127.0.98.1:34877)
I20260620 04:35:32.674110 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a77b2965333440f8590fb447e8d3358" candidate_uuid: "cd62d59a91994b94aeb355c5be11f66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b397d8f390492b962d280bda05b9d4"
I20260620 04:35:32.674177 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a77b2965333440f8590fb447e8d3358" candidate_uuid: "cd62d59a91994b94aeb355c5be11f66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573df7c27837422f94e310a6663f29c1"
I20260620 04:35:32.674299 1107 raft_consensus.cc:3037] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:32.674333 1375 raft_consensus.cc:3037] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:32.675374 1107 raft_consensus.cc:2454] T 0a77b2965333440f8590fb447e8d3358 P 573df7c27837422f94e310a6663f29c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd62d59a91994b94aeb355c5be11f66a in term 1.
I20260620 04:35:32.675515 1375 raft_consensus.cc:2454] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd62d59a91994b94aeb355c5be11f66a in term 1.
I20260620 04:35:32.675724 1176 leader_election.cc:302] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573df7c27837422f94e310a6663f29c1, cd62d59a91994b94aeb355c5be11f66a; no voters:
I20260620 04:35:32.675889 1445 raft_consensus.cc:2781] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:32.676152 1445 raft_consensus.cc:686] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [term 1 LEADER]: Becoming Leader. State: Replica: cd62d59a91994b94aeb355c5be11f66a, State: Running, Role: LEADER
I20260620 04:35:32.676483 1445 consensus_queue.cc:237] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } }
I20260620 04:35:32.679752 962 catalog_manager.cc:5495] T 0a77b2965333440f8590fb447e8d3358 P cd62d59a91994b94aeb355c5be11f66a reported cstate change: term changed from 0 to 1, leader changed from <none> to cd62d59a91994b94aeb355c5be11f66a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cd62d59a91994b94aeb355c5be11f66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b397d8f390492b962d280bda05b9d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd62d59a91994b94aeb355c5be11f66a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573df7c27837422f94e310a6663f29c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:32.705643 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.695940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.0.98.1:34877)
0620 04:35:32.696102 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.696109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.696309 (+ 200us) server_negotiation.cc:407] Connection header received
0620 04:35:32.696500 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.696503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.696548 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.696639 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.697385 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.697929 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.698908 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.699143 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.699269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.701963 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.701973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.701988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.702011 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.703866 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.704340 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.704347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.704349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.704432 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.704715 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.704718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.704720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.704971 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.705205 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.705349 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.705470 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:32.734210 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.725065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.98.1:34877)
0620 04:35:32.725351 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.725357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.725520 (+ 163us) server_negotiation.cc:407] Connection header received
0620 04:35:32.725705 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.725708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.725751 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.725836 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.726523 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.727029 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.727870 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.728095 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.728230 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.731122 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.731125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.731127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.731152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.732519 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.732971 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.732976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.732978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.733057 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.733343 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.733346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.733348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.733510 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.733642 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.733927 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.734043 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:32.761204 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:32.751493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.0.98.1:34877)
0620 04:35:32.751656 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:32.751663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:32.751976 (+ 313us) server_negotiation.cc:407] Connection header received
0620 04:35:32.752074 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:32.752076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:32.752121 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:32.752265 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:32.753100 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.753697 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.754626 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:32.754877 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:32.754974 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:32.757570 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:32.757573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:32.757575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:32.757599 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:32.759434 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.759872 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.759875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.759877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.759964 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:32.760300 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:32.760303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:32.760304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:32.760477 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:32.760594 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:32.760881 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 04:35:32.761017 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 04:35:32.762724 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0a77b2965333440f8590fb447e8d3358: Uninitialized: clean time has not yet been initialized
I20260620 04:35:32.764957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1024
W20260620 04:35:32.767064 1422 tablet.cc:2369] T 0a77b2965333440f8590fb447e8d3358 P b6b397d8f390492b962d280bda05b9d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:32.771559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1158
I20260620 04:35:32.779383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1292
I20260620 04:35:32.786458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 930
2026-06-20T04:35:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2423 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[1462](info): set up 2 sockets
Jun 20 04:35:32 dist-test-slave-zmf4 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 04:35:33.844308 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:33 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:33.962808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:34.076289 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:34.076679 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:34.076778 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:34.081007 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:34.081092 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:35:34.081122 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:34.081168 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:34.081219 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:34.081255 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:34.086565 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:34.087654 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:34.088709 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:34.098064 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:34.098205 1475 server_base.cc:1034] running on GCE node
W20260620 04:35:34.098248 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:34.098122 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:34.098927 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:34.100000 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:34.101202 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781930134101197 us; error 42 us; skew 500 ppm
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:34.103961 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:34.104918 1475 webserver.cc:460] Webserver started at http://127.0.98.62:41925/ using document root <none> and password file <none>
I20260620 04:35:34.105691 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:34.105815 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:34.106063 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:34.108505 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "691f21cc037e4072bc7555e9b27f51f7"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "a2595ef0471c50edcf060a1cf828337d"
server_key_iv: "2f38e93142dbae0bda569c63ef36a034"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.109073 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "691f21cc037e4072bc7555e9b27f51f7"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "a2595ef0471c50edcf060a1cf828337d"
server_key_iv: "2f38e93142dbae0bda569c63ef36a034"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.113204 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 04:35:34.115671 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:34.117045 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:35:34.117221 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "691f21cc037e4072bc7555e9b27f51f7"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "a2595ef0471c50edcf060a1cf828337d"
server_key_iv: "2f38e93142dbae0bda569c63ef36a034"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.117339 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:34.132829 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:34.136117 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:34.136376 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:34.145206 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35975
I20260620 04:35:34.145294 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35975 every 8 connection(s)
I20260620 04:35:34.146323 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:34.148727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1475
I20260620 04:35:34.148875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:34.149273 392 external_mini_cluster.cc:1428] Setting key 887374da6d367ac7e52c2036d2021957
I20260620 04:35:34.149351 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:34.156755 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Bootstrap starting.
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:34.159467 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:34.160432 1553 log.cc:824] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:34.162497 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: No bootstrap required, opened a new log
I20260620 04:35:34.165107 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.150786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.0.98.62:35975)
0620 04:35:34.151182 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:34.151201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:34.151240 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:35:34.152151 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:34.152174 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:34.152633 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:34.153065 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:34.154005 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.155495 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.156424 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.157117 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.157271 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.160032 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:34.160036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:34.160049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:34.160074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:34.162257 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.162747 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.162752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.162756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.162862 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.163203 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.163206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.163207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.163588 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:34.163746 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:34.164070 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 04:35:34.164416 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260620 04:35:34.166070 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } }
I20260620 04:35:34.166358 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:34.166421 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691f21cc037e4072bc7555e9b27f51f7, State: Initialized, Role: FOLLOWER
I20260620 04:35:34.166857 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } }
I20260620 04:35:34.167003 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:34.167068 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:34.167158 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:34.198915 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } }
I20260620 04:35:34.199462 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691f21cc037e4072bc7555e9b27f51f7; no voters:
I20260620 04:35:34.199891 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:34.200022 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:34.200297 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [term 1 LEADER]: Becoming Leader. State: Replica: 691f21cc037e4072bc7555e9b27f51f7, State: Running, Role: LEADER
I20260620 04:35:34.200655 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } }
I20260620 04:35:34.201442 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:34.202332 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691f21cc037e4072bc7555e9b27f51f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } } }
I20260620 04:35:34.202450 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:34.202854 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691f21cc037e4072bc7555e9b27f51f7. Latest consensus state: current_term: 1 leader_uuid: "691f21cc037e4072bc7555e9b27f51f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f21cc037e4072bc7555e9b27f51f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35975 } } }
I20260620 04:35:34.203001 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:34.204133 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:34.207489 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:34.213377 1566 catalog_manager.cc:1347] Generated new cluster ID: 339ad79698e1485c85550020ac2886cd
I20260620 04:35:34.213464 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:34.238528 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:34.239475 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:34.250797 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Generated new TSK 0
I20260620 04:35:34.251572 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260620 04:35:34.419021 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.154s user 0.000s sys 0.004s
I20260620 04:35:34.419492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35975
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:34.531801 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:34.532054 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:34.532099 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:34.536114 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:34.536181 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:34.536276 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:34.536350 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:34.541500 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:34.542546 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:34.543550 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:34.552877 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:34.552922 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:34.552922 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:34.553153 1579 server_base.cc:1034] running on GCE node
I20260620 04:35:34.553584 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:34.554219 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:34.555424 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781930134555398 us; error 48 us; skew 500 ppm
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:34.563396 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:34.564330 1579 webserver.cc:460] Webserver started at http://127.0.98.1:36419/ using document root <none> and password file <none>
I20260620 04:35:34.564921 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:34.565029 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:34.565264 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:34.567258 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "93d45c838a114e72a72f1aee8086ec78"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "473ea2ed485de207b88e3c783d264659"
server_key_iv: "78fd0847dd9462703f4763bd14d3a64c"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.567804 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93d45c838a114e72a72f1aee8086ec78"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "473ea2ed485de207b88e3c783d264659"
server_key_iv: "78fd0847dd9462703f4763bd14d3a64c"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.571877 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 04:35:34.574460 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:34.575846 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:35:34.576021 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "93d45c838a114e72a72f1aee8086ec78"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "473ea2ed485de207b88e3c783d264659"
server_key_iv: "78fd0847dd9462703f4763bd14d3a64c"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.576143 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:34.607044 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:34.610284 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:34.610502 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:34.611954 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:34.612028 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:34.612102 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:34.612160 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:34.623178 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42663
I20260620 04:35:34.623238 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42663 every 8 connection(s)
I20260620 04:35:34.624243 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:34.625296 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.612836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44777 (local address 127.0.98.62:35975)
0620 04:35:34.612976 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:34.612982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:34.613773 (+ 791us) server_negotiation.cc:407] Connection header received
0620 04:35:34.614840 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:34.614846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:34.614912 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:34.614976 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:34.616819 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.617421 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.618534 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.618768 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.618883 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.621505 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:34.621509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:34.621511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:34.621541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:34.623314 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.623770 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.623774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.623776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.623870 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.624180 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.624187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.624213 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.624430 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:34.624537 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:34.625075 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 04:35:34.625180 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:34.626016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1579
I20260620 04:35:34.626128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:34.626143 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.613177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.1:44777)
0620 04:35:34.613604 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:34.613651 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:34.614564 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:34.615439 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:34.615447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:34.615935 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:34.616672 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.616693 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.617548 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.617551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.618410 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.618418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.618941 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.618946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.619133 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.619742 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:34.619763 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:34.621340 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:34.623410 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.623415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.623425 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.623664 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.623965 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.623967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.623968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.624086 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.624526 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:34.624531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:34.624825 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 04:35:34.625036 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 04:35:34.625286 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
I20260620 04:35:34.626406 392 external_mini_cluster.cc:1428] Setting key 6d1488c76277c82d92a41652170c6c73
I20260620 04:35:34.627648 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35975
I20260620 04:35:34.627804 1710 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:34.628324 1710 heartbeater.cc:511] Master 127.0.98.62:35975 requested a full tablet report, sending...
I20260620 04:35:34.629817 1506 ts_manager.cc:194] Registered new tserver with Master: 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663)
I20260620 04:35:34.631111 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44777
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:34.638422 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.630204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48193 (local address 127.0.98.62:35975)
0620 04:35:34.630299 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:34.630303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:34.630316 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:35:34.630408 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:34.630411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:34.630449 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:34.630529 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:34.631069 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.631755 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.632969 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.633272 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.633349 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.634582 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:34.634586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:34.634589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:34.634610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:34.636749 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.637233 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.637237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.637239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.637326 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.637613 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.637617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.637618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.637788 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:34.637909 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:34.638154 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 04:35:34.638266 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:35:34.638434 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.629903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.1:48193)
0620 04:35:34.630207 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:34.630225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:34.630323 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:34.630566 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:34.630570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:34.630797 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:34.630941 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.630951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.631885 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.631887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.632833 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.632843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.633493 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.633496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.633587 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.634119 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:34.634130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:34.634441 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:34.636864 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.636869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.636871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.637136 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.637425 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.637428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.637430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.637516 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.637887 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:34.637890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:34.637975 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 04:35:34.638091 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 04:35:34.638233 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":18,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:34.655810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35975
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:34.765151 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:34.765404 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:34.765447 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:34.769491 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:34.769549 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:34.769590 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:34.769622 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:34.774724 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:34.775640 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:34.776799 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:34.786541 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:34.786677 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:34.786522 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:34.786580 1719 server_base.cc:1034] running on GCE node
I20260620 04:35:34.787115 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:34.787652 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:34.788820 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781930134788783 us; error 47 us; skew 500 ppm
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:34.791551 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:34.792496 1719 webserver.cc:460] Webserver started at http://127.0.98.2:46595/ using document root <none> and password file <none>
I20260620 04:35:34.793085 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:34.793184 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:34.793421 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:34.795524 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "841e6124331140eea5e45ac9e88cabf8"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "980d2ffb6b04d4434f5fbdb5a8c504d4"
server_key_iv: "c4ba211879250fc84983ad31cb5cae9f"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.796129 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "841e6124331140eea5e45ac9e88cabf8"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "980d2ffb6b04d4434f5fbdb5a8c504d4"
server_key_iv: "c4ba211879250fc84983ad31cb5cae9f"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.799964 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:34.802677 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:34.803968 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:34.804107 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "841e6124331140eea5e45ac9e88cabf8"
format_stamp: "Formatted at 2026-06-20 04:35:34 on dist-test-slave-zmf4"
server_key: "980d2ffb6b04d4434f5fbdb5a8c504d4"
server_key_iv: "c4ba211879250fc84983ad31cb5cae9f"
server_key_version: "encryptionkey@0"
I20260620 04:35:34.804244 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:34.833456 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:34.836824 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:34.837021 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:34.838868 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:34.838932 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:34.838985 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:34.839016 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:34 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:34.850450 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33109
I20260620 04:35:34.850724 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33109 every 8 connection(s)
I20260620 04:35:34.851528 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:34.851933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1719
I20260620 04:35:34.852043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:34.852383 392 external_mini_cluster.cc:1428] Setting key b22705d1412efe696575979f82ef2efe
I20260620 04:35:34.853824 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.839744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53663 (local address 127.0.98.62:35975)
0620 04:35:34.839921 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:34.839929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:34.840838 (+ 909us) server_negotiation.cc:407] Connection header received
0620 04:35:34.842068 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:34.842072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:34.842136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:34.842259 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:34.843630 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.844518 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.845657 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.845890 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.846003 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.849508 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:34.849512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:34.849515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:34.849549 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:34.851256 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.851983 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.851988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.851989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.852083 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.852518 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.852522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.852524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.852765 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:34.852865 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:34.853470 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 04:35:34.853637 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:35:34.854549 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.840114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.2:53663)
0620 04:35:34.840655 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:34.840714 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:34.841849 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:34.842433 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:34.842442 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:34.842922 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:34.843470 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.843490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.844655 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.844659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.845529 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.845538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.846054 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.846057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.846265 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.847044 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:34.847060 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:34.849317 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:34.851413 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.851418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.851429 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.851854 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.852223 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.852226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.852228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.852403 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.852882 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:34.852891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:34.853173 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 04:35:34.853361 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 04:35:34.853703 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":120,"threads_started":1}
I20260620 04:35:34.856580 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35975
I20260620 04:35:34.856767 1850 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:34.857287 1850 heartbeater.cc:511] Master 127.0.98.62:35975 requested a full tablet report, sending...
I20260620 04:35:34.858356 1507 ts_manager.cc:194] Registered new tserver with Master: 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2:33109)
I20260620 04:35:34.859016 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53663
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:34.866961 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.858988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.2:60879)
0620 04:35:34.859478 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:34.859495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:34.859636 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:34.859934 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:34.859939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:34.860146 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:34.860301 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.860311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.861354 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.861359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.862459 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:34.862474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.862951 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.862954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:34.863023 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.863533 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:34.863548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:34.863870 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:34.865362 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.865364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.865366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.865606 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.865884 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:34.865891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:34.865895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:34.865996 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:34.866378 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:34.866383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:34.866493 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 04:35:34.866611 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 04:35:34.866769 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":233,"threads_started":1}
I20260620 04:35:34.867081 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:34.859046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60879 (local address 127.0.98.62:35975)
0620 04:35:34.859178 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:34.859184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:34.859562 (+ 378us) server_negotiation.cc:407] Connection header received
0620 04:35:34.859753 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:34.859757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:34.859804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:34.859896 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:34.860430 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.861217 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.862605 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:34.862840 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:34.862924 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:34.863987 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:34.863990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:34.863991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:34.864011 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:34.865238 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.865715 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.865718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.865719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.865792 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:34.866085 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:34.866087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:34.866088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:34.866284 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:34.866360 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:34.866749 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 04:35:34.866885 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:34.883484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35975
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:34.993058 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:34.993309 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:34.993353 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:34.997308 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:34.997362 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:34.997401 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:34.997432 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:35.003291 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:35.004387 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:35.005407 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:35.015293 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:35.015194 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:35.015312 1859 server_base.cc:1034] running on GCE node
W20260620 04:35:35.015201 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:35.015901 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:35.016515 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:35.017709 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781930135017677 us; error 41 us; skew 500 ppm
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:35.020462 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:35.021335 1859 webserver.cc:460] Webserver started at http://127.0.98.3:37001/ using document root <none> and password file <none>
I20260620 04:35:35.021867 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:35.021924 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:35.022121 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:35.024019 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "e5d67bfc9d8149529c862cae7ec1f97d"
format_stamp: "Formatted at 2026-06-20 04:35:35 on dist-test-slave-zmf4"
server_key: "ccf69f5c6f8231bb691c24836c032a1e"
server_key_iv: "c565e2674bdf752c1574a7a3bfc2b6e7"
server_key_version: "encryptionkey@0"
I20260620 04:35:35.024614 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e5d67bfc9d8149529c862cae7ec1f97d"
format_stamp: "Formatted at 2026-06-20 04:35:35 on dist-test-slave-zmf4"
server_key: "ccf69f5c6f8231bb691c24836c032a1e"
server_key_iv: "c565e2674bdf752c1574a7a3bfc2b6e7"
server_key_version: "encryptionkey@0"
I20260620 04:35:35.028347 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:35.030939 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:35.032094 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:35.032258 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "e5d67bfc9d8149529c862cae7ec1f97d"
format_stamp: "Formatted at 2026-06-20 04:35:35 on dist-test-slave-zmf4"
server_key: "ccf69f5c6f8231bb691c24836c032a1e"
server_key_iv: "c565e2674bdf752c1574a7a3bfc2b6e7"
server_key_version: "encryptionkey@0"
I20260620 04:35:35.032361 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:35.054468 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:35.057772 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:35.057972 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:35.059736 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:35.059791 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:35.059859 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:35.059896 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:35.071353 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37153
I20260620 04:35:35.072077 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37153 every 8 connection(s)
I20260620 04:35:35.072183 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:35.074476 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.060559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57491 (local address 127.0.98.62:35975)
0620 04:35:35.060691 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.060695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.061763 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 04:35:35.062654 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.062658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.062713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.062815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.064078 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.064751 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.065878 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.066238 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.066373 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.070267 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.070271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.070274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.070310 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.071864 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.072467 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.072473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.072478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.072570 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.073105 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.073107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.073109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.073293 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.073402 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.074161 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.074294 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:35.075243 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.061137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.3:57491)
0620 04:35:35.061624 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.061663 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.062450 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.062985 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.062994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.063479 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:35.063932 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.063945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.064873 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.064879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.065749 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.065758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.066403 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.066410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.066656 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.067281 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:35.067304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:35.070065 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:35.071985 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.071991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.072007 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.072365 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.072714 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.072717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.072719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.072868 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.073398 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:35.073404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:35.073873 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.074125 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.074442 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
I20260620 04:35:35.076596 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35975
I20260620 04:35:35.076754 1990 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:35.077270 1990 heartbeater.cc:511] Master 127.0.98.62:35975 requested a full tablet report, sending...
I20260620 04:35:35.078334 1506 ts_manager.cc:194] Registered new tserver with Master: e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153)
I20260620 04:35:35.078990 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57491
I20260620 04:35:35.079797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 1859
I20260620 04:35:35.079931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:35.080231 392 external_mini_cluster.cc:1428] Setting key e6dcb57645a81b9143360ea946290034
I20260620 04:35:35.082170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:35:35.086902 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.078771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50211 (local address 127.0.98.62:35975)
0620 04:35:35.078905 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.078910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.079076 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:35:35.079246 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.079251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.079300 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.079395 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.079861 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.080604 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.081788 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.082080 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.082198 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.083252 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.083259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.083265 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.083301 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.085135 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.085691 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.085693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.085695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.085775 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.086141 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.086143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.086144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.086294 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.086369 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.086674 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.086761 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:35:35.087100 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.078704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.98.3:50211)
0620 04:35:35.078997 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.079008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.079114 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.079413 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.079416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.079612 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:35.079726 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.079733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.080870 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.080874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.081665 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.081675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.082189 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.082192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.082320 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.082811 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:35.082822 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:35.083119 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:35.085255 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.085259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.085261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.085566 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.085929 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.085935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.085940 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.086044 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.086389 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:35.086395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:35.086505 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.086622 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.086787 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930135, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930135, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:35.110358 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.101482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.62:35975)
0620 04:35:35.101654 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.101658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.101671 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:35:35.101706 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.101708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.101743 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.101821 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.102656 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.103240 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.104121 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.104394 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.104502 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.106940 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.106944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.106947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.106981 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.108483 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.109107 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.109112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.109115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.109228 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.109507 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.109512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.109513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.109696 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.109833 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.110068 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.110189 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 04:35:35.113269 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:35.116302 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:35.130978 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.125429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33109 (local address 127.0.0.1:59432)
0620 04:35:35.126357 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.126371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.126451 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.127185 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.127188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.127206 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.127312 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.127318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.128690 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.128693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.129601 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.129609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.130434 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.130438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.130548 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.130587 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.130727 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.130841 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":85,"threads_started":1}
I20260620 04:35:35.130990 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.125748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.98.2:33109)
0620 04:35:35.126172 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.126183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.126549 (+ 366us) server_negotiation.cc:407] Connection header received
0620 04:35:35.126635 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.126644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.126866 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.127045 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.127441 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.128549 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.129731 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.130312 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.130436 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.130600 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.130755 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.130843 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":121,"threads_started":1}
I20260620 04:35:35.131842 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.124927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37153 (local address 127.0.0.1:47360)
0620 04:35:35.125243 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.125254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.125395 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.126698 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.126702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.126719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.126815 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.126819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.128231 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.128235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.130400 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.130405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.131466 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.131470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.131559 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.131579 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.131632 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.131694 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":320,"spinlock_wait_cycles":3712,"thread_start_us":93,"threads_started":1}
I20260620 04:35:35.132025 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.124733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42663 (local address 127.0.0.1:33858)
0620 04:35:35.125085 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.125137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.125396 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.126366 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.126374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.126403 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.126560 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.126569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.128068 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.128075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.130394 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.130404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.131660 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.131663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.131750 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.131771 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.131840 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.131904 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":264,"spinlock_wait_cycles":6656,"thread_start_us":77,"threads_started":1}
I20260620 04:35:35.132444 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.124898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.98.1:42663)
0620 04:35:35.125784 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.125803 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.125829 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:35:35.125914 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.125922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.126149 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.126725 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.126863 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.127949 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.130539 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.131329 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.131463 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.131591 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.132214 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.132325 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":87,"threads_started":1}
I20260620 04:35:35.132498 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.125840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.0.98.3:37153)
0620 04:35:35.126197 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.126209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.126230 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:35:35.126302 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.126308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.126496 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.126924 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.127060 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.128072 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.130547 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.131329 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.132011 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.132168 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.132245 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.132349 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":95,"threads_started":1}
I20260620 04:35:35.134275 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3cd50b5f47419d986d799f29519e75 (DEFAULT_TABLE table=test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.135138 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3cd50b5f47419d986d799f29519e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.135680 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3cd50b5f47419d986d799f29519e75 (DEFAULT_TABLE table=test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.135680 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3cd50b5f47419d986d799f29519e75 (DEFAULT_TABLE table=test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.136680 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3cd50b5f47419d986d799f29519e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.136673 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3cd50b5f47419d986d799f29519e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.142066 2013 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Bootstrap starting.
I20260620 04:35:35.143901 2014 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Bootstrap starting.
I20260620 04:35:35.144443 2013 tablet_bootstrap.cc:654] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.144645 2015 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Bootstrap starting.
I20260620 04:35:35.145325 2013 log.cc:824] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:35.146242 2014 tablet_bootstrap.cc:654] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.146472 2015 tablet_bootstrap.cc:654] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.147050 2014 log.cc:824] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:35.147143 2015 log.cc:824] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:35.147364 2013 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: No bootstrap required, opened a new log
I20260620 04:35:35.147527 2013 ts_tablet_manager.cc:1397] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 04:35:35.148840 2015 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: No bootstrap required, opened a new log
I20260620 04:35:35.148998 2015 ts_tablet_manager.cc:1397] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260620 04:35:35.149397 2014 tablet_bootstrap.cc:492] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: No bootstrap required, opened a new log
I20260620 04:35:35.149546 2014 ts_tablet_manager.cc:1397] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 04:35:35.151298 2013 raft_consensus.cc:348] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.151527 2013 raft_consensus.cc:374] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.151566 2013 raft_consensus.cc:729] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841e6124331140eea5e45ac9e88cabf8, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.151978 2013 consensus_queue.cc:260] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.153450 2013 ts_tablet_manager.cc:1428] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 04:35:35.153841 1850 heartbeater.cc:503] Master 127.0.98.62:35975 was elected leader, sending a full tablet report...
I20260620 04:35:35.153949 2014 raft_consensus.cc:348] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.154547 2014 raft_consensus.cc:374] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.154613 2014 raft_consensus.cc:729] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d45c838a114e72a72f1aee8086ec78, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.155046 2014 consensus_queue.cc:260] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.155157 2015 raft_consensus.cc:348] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.155485 2015 raft_consensus.cc:374] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.155575 2015 raft_consensus.cc:729] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d67bfc9d8149529c862cae7ec1f97d, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.155956 2014 ts_tablet_manager.cc:1428] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:35:35.156070 2015 consensus_queue.cc:260] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.156744 1990 heartbeater.cc:503] Master 127.0.98.62:35975 was elected leader, sending a full tablet report...
I20260620 04:35:35.157100 2015 ts_tablet_manager.cc:1428] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260620 04:35:35.157200 1710 heartbeater.cc:503] Master 127.0.98.62:35975 was elected leader, sending a full tablet report...
I20260620 04:35:35.296623 2019 raft_consensus.cc:482] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:35.296840 2019 raft_consensus.cc:504] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.298048 2019 leader_election.cc:288] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153)
I20260620 04:35:35.302320 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.298423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50841 (local address 127.0.98.1:42663)
0620 04:35:35.298544 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.298550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.298945 (+ 395us) server_negotiation.cc:407] Connection header received
0620 04:35:35.299104 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.299107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.299161 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.299252 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.299519 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.300102 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301242 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.301736 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.301918 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.302095 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.302198 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:35:35.302382 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.298424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46445 (local address 127.0.98.3:37153)
0620 04:35:35.298552 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.298556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.298570 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:35:35.298720 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.298728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.298823 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.298902 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.299160 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.299764 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301382 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.301823 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301929 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.301986 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.302204 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.302275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:35:35.302388 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.298338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37153 (local address 127.0.98.2:46445)
0620 04:35:35.298469 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.298486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.298579 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.298914 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.298917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.298936 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.299056 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.299061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.299895 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.299899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.301205 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.301214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.301940 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.302070 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.302092 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.302175 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.302256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 04:35:35.302312 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.298349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42663 (local address 127.0.98.2:50841)
0620 04:35:35.298852 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.298863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.298990 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.299252 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.299254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.299269 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.299412 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.299417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.300230 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.300234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.301112 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.301120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.301856 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.301859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.301963 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.301985 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.302065 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.302148 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":110,"spinlock_wait_cycles":896,"thread_start_us":144,"threads_started":1}
I20260620 04:35:35.303048 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3cd50b5f47419d986d799f29519e75" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" is_pre_election: true
I20260620 04:35:35.303049 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3cd50b5f47419d986d799f29519e75" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78" is_pre_election: true
I20260620 04:35:35.303347 1942 raft_consensus.cc:2454] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 0.
I20260620 04:35:35.303347 1664 raft_consensus.cc:2454] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 0.
I20260620 04:35:35.303804 1735 leader_election.cc:302] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, 93d45c838a114e72a72f1aee8086ec78, e5d67bfc9d8149529c862cae7ec1f97d; no voters:
I20260620 04:35:35.304075 2019 raft_consensus.cc:2781] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:35.304145 2019 raft_consensus.cc:482] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:35.304198 2019 raft_consensus.cc:3037] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.305259 2019 raft_consensus.cc:504] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.305632 2019 leader_election.cc:288] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 election: Requested vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153)
I20260620 04:35:35.305962 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3cd50b5f47419d986d799f29519e75" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67bfc9d8149529c862cae7ec1f97d"
I20260620 04:35:35.306016 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3cd50b5f47419d986d799f29519e75" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78"
I20260620 04:35:35.306093 1942 raft_consensus.cc:3037] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.306156 1664 raft_consensus.cc:3037] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.307156 1942 raft_consensus.cc:2454] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 1.
I20260620 04:35:35.307346 1664 raft_consensus.cc:2454] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 1.
I20260620 04:35:35.307494 1738 leader_election.cc:302] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, e5d67bfc9d8149529c862cae7ec1f97d; no voters:
I20260620 04:35:35.307763 2019 raft_consensus.cc:2781] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:35.307958 2019 raft_consensus.cc:686] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 1 LEADER]: Becoming Leader. State: Replica: 841e6124331140eea5e45ac9e88cabf8, State: Running, Role: LEADER
I20260620 04:35:35.308321 2019 consensus_queue.cc:237] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } }
I20260620 04:35:35.311864 1507 catalog_manager.cc:5495] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "841e6124331140eea5e45ac9e88cabf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } health_report { overall_health: HEALTHY } } }
W20260620 04:35:35.323432 1991 tablet.cc:2369] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:35.330714 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.326761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.98.2:33109)
0620 04:35:35.326916 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.326919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.326933 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:35:35.327105 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.327108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.327171 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.327259 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:35.327584 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.328101 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.329044 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.329285 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.329366 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.329849 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.330223 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.330314 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.330545 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.330594 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:35:35.336339 1664 raft_consensus.cc:1261] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Refusing update from remote peer 841e6124331140eea5e45ac9e88cabf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.336380 1942 raft_consensus.cc:1261] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Refusing update from remote peer 841e6124331140eea5e45ac9e88cabf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.337074 2023 consensus_queue.cc:1035] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.337240 2019 consensus_queue.cc:1035] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.343562 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298785834329452544 to 7298785834225418240. Current Snapshot: MvccSnapshot[applied={T|T < 7298785834329452544}]
I20260620 04:35:35.355414 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:35.355733 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:35.358286 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Sending DeleteTablet for 3 replicas of tablet 6b3cd50b5f47419d986d799f29519e75
I20260620 04:35:35.359012 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3cd50b5f47419d986d799f29519e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33858
I20260620 04:35:35.359206 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3cd50b5f47419d986d799f29519e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59432
I20260620 04:35:35.359539 2036 tablet_replica.cc:330] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: stopping tablet replica
I20260620 04:35:35.359644 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3cd50b5f47419d986d799f29519e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47360
I20260620 04:35:35.359843 2037 tablet_replica.cc:330] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: stopping tablet replica
I20260620 04:35:35.360001 2036 raft_consensus.cc:2229] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:35.360092 2037 raft_consensus.cc:2229] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:35.360097 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:35:35.360504 2037 raft_consensus.cc:2258] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:35:35.360522 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:35.360556 2036 raft_consensus.cc:2258] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:35.362022 2036 ts_tablet_manager.cc:1905] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.362317 2037 ts_tablet_manager.cc:1905] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.364795 2036 ts_tablet_manager.cc:1918] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:35.364951 2036 log.cc:1196] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/6b3cd50b5f47419d986d799f29519e75
I20260620 04:35:35.364944 2037 ts_tablet_manager.cc:1918] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:35.365023 2037 log.cc:1196] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/6b3cd50b5f47419d986d799f29519e75
I20260620 04:35:35.365350 2037 ts_tablet_manager.cc:1939] T 6b3cd50b5f47419d986d799f29519e75 P 841e6124331140eea5e45ac9e88cabf8: Deleting consensus metadata
I20260620 04:35:35.365649 2036 ts_tablet_manager.cc:1939] T 6b3cd50b5f47419d986d799f29519e75 P 93d45c838a114e72a72f1aee8086ec78: Deleting consensus metadata
I20260620 04:35:35.366139 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (DEFAULT_TABLE table=test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.366389 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.366559 2038 tablet_replica.cc:330] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: stopping tablet replica
I20260620 04:35:35.366853 2038 raft_consensus.cc:2229] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:35.367130 2038 raft_consensus.cc:2258] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:35.367821 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (DEFAULT_TABLE table=test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.368060 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.368435 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (DEFAULT_TABLE table=test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:35.368664 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.368662 2014 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Bootstrap starting.
I20260620 04:35:35.369612 2014 tablet_bootstrap.cc:654] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.370388 1491 catalog_manager.cc:4841] TS 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663): tablet 6b3cd50b5f47419d986d799f29519e75 (table test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]) successfully deleted
I20260620 04:35:35.370539 2015 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Bootstrap starting.
I20260620 04:35:35.370939 1494 catalog_manager.cc:4841] TS 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2:33109): tablet 6b3cd50b5f47419d986d799f29519e75 (table test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]) successfully deleted
I20260620 04:35:35.371415 2014 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: No bootstrap required, opened a new log
I20260620 04:35:35.371423 2015 tablet_bootstrap.cc:654] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.371505 2014 ts_tablet_manager.cc:1397] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:35.371964 2013 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Bootstrap starting.
I20260620 04:35:35.372117 2014 raft_consensus.cc:348] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.372251 2014 raft_consensus.cc:374] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.372291 2014 raft_consensus.cc:729] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d45c838a114e72a72f1aee8086ec78, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.372423 2014 consensus_queue.cc:260] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.372664 2014 ts_tablet_manager.cc:1428] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:35.372872 2038 ts_tablet_manager.cc:1905] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.373081 2013 tablet_bootstrap.cc:654] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.374146 2015 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: No bootstrap required, opened a new log
I20260620 04:35:35.374226 2015 ts_tablet_manager.cc:1397] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 04:35:35.374622 2015 raft_consensus.cc:348] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.374737 2015 raft_consensus.cc:374] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.374773 2015 raft_consensus.cc:729] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d67bfc9d8149529c862cae7ec1f97d, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.374881 2015 consensus_queue.cc:260] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.375129 2015 ts_tablet_manager.cc:1428] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260620 04:35:35.375586 1711 tablet.cc:2369] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:35.376245 2038 ts_tablet_manager.cc:1918] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:35.376480 2038 log.cc:1196] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/6b3cd50b5f47419d986d799f29519e75
I20260620 04:35:35.377059 2038 ts_tablet_manager.cc:1939] T 6b3cd50b5f47419d986d799f29519e75 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting consensus metadata
I20260620 04:35:35.377954 2013 tablet_bootstrap.cc:492] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: No bootstrap required, opened a new log
I20260620 04:35:35.378047 2013 ts_tablet_manager.cc:1397] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 04:35:35.378510 1494 catalog_manager.cc:4841] TS e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153): tablet 6b3cd50b5f47419d986d799f29519e75 (table test-table [id=f162ae6a29f84ba4a31b1dae5b75b992]) successfully deleted
I20260620 04:35:35.378496 2013 raft_consensus.cc:348] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.378600 2013 raft_consensus.cc:374] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.378638 2013 raft_consensus.cc:729] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841e6124331140eea5e45ac9e88cabf8, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.378764 2013 consensus_queue.cc:260] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.379024 2013 ts_tablet_manager.cc:1428] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:35.584853 2033 raft_consensus.cc:482] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:35.584985 2033 raft_consensus.cc:504] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.585383 2033 leader_election.cc:288] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153)
I20260620 04:35:35.585858 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8739eb1fdbfc48ad9cd6b1c373fa2f61" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" is_pre_election: true
I20260620 04:35:35.585896 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8739eb1fdbfc48ad9cd6b1c373fa2f61" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78" is_pre_election: true
I20260620 04:35:35.586032 1942 raft_consensus.cc:2454] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 0.
I20260620 04:35:35.586086 1664 raft_consensus.cc:2454] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 0.
I20260620 04:35:35.586400 1738 leader_election.cc:302] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, e5d67bfc9d8149529c862cae7ec1f97d; no voters:
I20260620 04:35:35.586614 2033 raft_consensus.cc:2781] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:35.586678 2033 raft_consensus.cc:482] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:35.586735 2033 raft_consensus.cc:3037] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.587759 2033 raft_consensus.cc:504] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.588109 2033 leader_election.cc:288] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 election: Requested vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153)
I20260620 04:35:35.588557 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8739eb1fdbfc48ad9cd6b1c373fa2f61" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67bfc9d8149529c862cae7ec1f97d"
I20260620 04:35:35.588550 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8739eb1fdbfc48ad9cd6b1c373fa2f61" candidate_uuid: "841e6124331140eea5e45ac9e88cabf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78"
I20260620 04:35:35.588685 1664 raft_consensus.cc:3037] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.588698 1942 raft_consensus.cc:3037] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.589855 1664 raft_consensus.cc:2454] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 1.
I20260620 04:35:35.589855 1942 raft_consensus.cc:2454] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 841e6124331140eea5e45ac9e88cabf8 in term 1.
I20260620 04:35:35.590165 1735 leader_election.cc:302] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, 93d45c838a114e72a72f1aee8086ec78; no voters:
I20260620 04:35:35.590402 2033 raft_consensus.cc:2781] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:35.590524 2033 raft_consensus.cc:686] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 1 LEADER]: Becoming Leader. State: Replica: 841e6124331140eea5e45ac9e88cabf8, State: Running, Role: LEADER
I20260620 04:35:35.590691 2033 consensus_queue.cc:237] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.592492 1505 catalog_manager.cc:5495] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "841e6124331140eea5e45ac9e88cabf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } health_report { overall_health: UNKNOWN } } }
W20260620 04:35:35.603430 1851 tablet.cc:2369] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:35.638918 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.629707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35975 (local address 127.0.0.1:42384)
0620 04:35:35.630007 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.630020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.630119 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.630377 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.630379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.630726 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:35.630824 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.630830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.631863 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.631866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.632569 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.632578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.633154 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.633157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.633217 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.633820 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:35.633832 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:35.635439 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:35.637447 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.637455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.637460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.637693 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.637963 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:35.637965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:35.637967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:35.638043 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:35.638438 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:35.638443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:35.638537 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.638629 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.638738 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260620 04:35:35.638921 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.629800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.0.98.62:35975)
0620 04:35:35.630054 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.630062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.630077 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:35:35.630213 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.630216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.630259 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.630349 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.630929 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.631754 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.632681 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.633033 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.633127 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.635567 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.635571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.635573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.635596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.637318 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.637788 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.637791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.637793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.637876 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.638129 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.638132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.638134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.638334 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.638438 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.638694 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.638789 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
I20260620 04:35:35.642966 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42384:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:35:35.643712 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:35.648598 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 608d9b99b7aa49b8b1c23845809bbcff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f00c07048e41298d30e06ebb61658d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:35.648607 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 608d9b99b7aa49b8b1c23845809bbcff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f00c07048e41298d30e06ebb61658d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:35.648607 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 608d9b99b7aa49b8b1c23845809bbcff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f00c07048e41298d30e06ebb61658d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:35.648972 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608d9b99b7aa49b8b1c23845809bbcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.648972 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608d9b99b7aa49b8b1c23845809bbcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.648988 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608d9b99b7aa49b8b1c23845809bbcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:35.651505 2014 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: Bootstrap starting.
I20260620 04:35:35.651865 2013 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: Bootstrap starting.
I20260620 04:35:35.652060 2015 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: Bootstrap starting.
I20260620 04:35:35.652554 2014 tablet_bootstrap.cc:654] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.652915 2013 tablet_bootstrap.cc:654] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.653028 2015 tablet_bootstrap.cc:654] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:35.654138 2013 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: No bootstrap required, opened a new log
I20260620 04:35:35.654145 2014 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: No bootstrap required, opened a new log
I20260620 04:35:35.654243 2014 ts_tablet_manager.cc:1397] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:35.654247 2013 ts_tablet_manager.cc:1397] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:35.654268 2015 tablet_bootstrap.cc:492] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: No bootstrap required, opened a new log
I20260620 04:35:35.654336 2015 ts_tablet_manager.cc:1397] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:35.654738 2015 raft_consensus.cc:348] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.654716 2014 raft_consensus.cc:348] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.654717 2013 raft_consensus.cc:348] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.654837 2014 raft_consensus.cc:374] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.654836 2015 raft_consensus.cc:374] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.654838 2013 raft_consensus.cc:374] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:35.654870 2015 raft_consensus.cc:729] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d67bfc9d8149529c862cae7ec1f97d, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.654870 2014 raft_consensus.cc:729] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d45c838a114e72a72f1aee8086ec78, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.654883 2013 raft_consensus.cc:729] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841e6124331140eea5e45ac9e88cabf8, State: Initialized, Role: FOLLOWER
I20260620 04:35:35.654990 2015 consensus_queue.cc:260] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.654991 2014 consensus_queue.cc:260] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.655009 2013 consensus_queue.cc:260] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.655246 2015 ts_tablet_manager.cc:1428] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:35.655256 2014 ts_tablet_manager.cc:1428] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:35.655262 2013 ts_tablet_manager.cc:1428] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:35.655433 2021 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.655463 2020 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.655578 2021 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.655578 2020 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.655681 2033 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.655813 2033 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.726464 2021 raft_consensus.cc:482] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:35.726603 2021 raft_consensus.cc:504] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.727958 2021 leader_election.cc:288] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2:33109)
I20260620 04:35:35.731988 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.728348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34907 (local address 127.0.98.2:33109)
0620 04:35:35.728532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.728537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.728990 (+ 453us) server_negotiation.cc:407] Connection header received
0620 04:35:35.729139 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.729142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.729190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.729272 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.729518 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.730005 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731079 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.731464 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.731654 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.731811 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.731874 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 04:35:35.732013 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.728269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33109 (local address 127.0.98.3:34907)
0620 04:35:35.728900 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.728912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.729020 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.729282 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.729284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.729299 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.729413 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.729419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.730122 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.730126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.730960 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.730968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.731578 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.731681 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.731702 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.731796 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.731860 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":402,"thread_start_us":93,"threads_started":1}
I20260620 04:35:35.732125 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.728348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38017 (local address 127.0.98.1:42663)
0620 04:35:35.728659 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.728666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.728683 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:35:35.728880 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.728884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.728937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.729027 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:35.729320 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.729883 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731088 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.731516 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731645 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.731706 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.731904 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.731976 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260620 04:35:35.732393 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.728265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42663 (local address 127.0.98.3:38017)
0620 04:35:35.728610 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.728627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.728754 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.729042 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.729045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.729064 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:35.729197 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.729204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.730002 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.730008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.730951 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.730960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.731645 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.731649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.731767 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.731794 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.731889 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.731961 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260620 04:35:35.732575 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608d9b99b7aa49b8b1c23845809bbcff" candidate_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841e6124331140eea5e45ac9e88cabf8" is_pre_election: true
I20260620 04:35:35.732860 1804 raft_consensus.cc:2454] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d67bfc9d8149529c862cae7ec1f97d in term 0.
I20260620 04:35:35.732901 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608d9b99b7aa49b8b1c23845809bbcff" candidate_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78" is_pre_election: true
I20260620 04:35:35.733062 1664 raft_consensus.cc:2454] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d67bfc9d8149529c862cae7ec1f97d in term 0.
I20260620 04:35:35.733320 1878 leader_election.cc:302] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, e5d67bfc9d8149529c862cae7ec1f97d; no voters:
I20260620 04:35:35.733585 2021 raft_consensus.cc:2781] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:35.733657 2021 raft_consensus.cc:482] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:35.733685 2021 raft_consensus.cc:3037] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.734660 2021 raft_consensus.cc:504] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.735023 2021 leader_election.cc:288] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [CANDIDATE]: Term 1 election: Requested vote from peers 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663), 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2:33109)
I20260620 04:35:35.735327 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608d9b99b7aa49b8b1c23845809bbcff" candidate_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93d45c838a114e72a72f1aee8086ec78"
I20260620 04:35:35.735391 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608d9b99b7aa49b8b1c23845809bbcff" candidate_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841e6124331140eea5e45ac9e88cabf8"
I20260620 04:35:35.735436 1664 raft_consensus.cc:3037] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.735539 1804 raft_consensus.cc:3037] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:35.736382 1664 raft_consensus.cc:2454] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d67bfc9d8149529c862cae7ec1f97d in term 1.
I20260620 04:35:35.736744 1804 raft_consensus.cc:2454] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d67bfc9d8149529c862cae7ec1f97d in term 1.
I20260620 04:35:35.737056 1878 leader_election.cc:302] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 841e6124331140eea5e45ac9e88cabf8, e5d67bfc9d8149529c862cae7ec1f97d; no voters:
I20260620 04:35:35.737254 2021 raft_consensus.cc:2781] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:35.737423 2021 raft_consensus.cc:686] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [term 1 LEADER]: Becoming Leader. State: Replica: e5d67bfc9d8149529c862cae7ec1f97d, State: Running, Role: LEADER
I20260620 04:35:35.737671 2021 consensus_queue.cc:237] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } }
I20260620 04:35:35.739327 2055 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: TxnStatusTablet state changed. Reason: New leader e5d67bfc9d8149529c862cae7ec1f97d. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.739431 2055 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:35.739665 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:35:35.739828 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:35:35.740142 1506 catalog_manager.cc:5495] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } health_report { overall_health: HEALTHY } } }
I20260620 04:35:35.757186 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.752761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37153 (local address 127.0.0.1:49310)
0620 04:35:35.752977 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.752991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.753083 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.753887 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.753890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.753900 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:35.754005 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.754012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.754706 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.754710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.755657 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.755674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.756090 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.756094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.756182 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.756262 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.756863 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.756876 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.756947 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.757031 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 04:35:35.757200 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.753526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.0.98.3:37153)
0620 04:35:35.753646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.753649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.753661 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:35:35.753713 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.753715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.753768 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.753849 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:35.754110 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.754604 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.755775 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.755968 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.756080 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.756384 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.756756 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.756839 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.756972 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.757026 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:35:35.760027 1664 raft_consensus.cc:1261] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Refusing update from remote peer e5d67bfc9d8149529c862cae7ec1f97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.760260 1804 raft_consensus.cc:1261] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Refusing update from remote peer e5d67bfc9d8149529c862cae7ec1f97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.760484 2021 consensus_queue.cc:1035] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.760712 2055 consensus_queue.cc:1035] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.761962 2020 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: TxnStatusTablet state changed. Reason: New leader e5d67bfc9d8149529c862cae7ec1f97d. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.762080 2020 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.762187 2033 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: TxnStatusTablet state changed. Reason: New leader e5d67bfc9d8149529c862cae7ec1f97d. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.762302 2033 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.764974 2055 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.765074 2055 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:35.766041 2055 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.766147 2055 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P e5d67bfc9d8149529c862cae7ec1f97d: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:35.766314 2033 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.766420 2033 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 841e6124331140eea5e45ac9e88cabf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.766500 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298785836067995648 to 7298785835983843328. Current Snapshot: MvccSnapshot[applied={T|T < 7298785836067995648 or (T in {7298785836067995648})}]
I20260620 04:35:35.766594 2020 tablet_replica.cc:439] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 } } peers { permanent_uuid: "841e6124331140eea5e45ac9e88cabf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33109 } } peers { permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 } } }
I20260620 04:35:35.766686 2020 tablet_replica.cc:442] T 608d9b99b7aa49b8b1c23845809bbcff P 93d45c838a114e72a72f1aee8086ec78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:35.782511 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.778353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54973 (local address 127.0.98.3:37153)
0620 04:35:35.778476 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.778481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.778852 (+ 371us) server_negotiation.cc:407] Connection header received
0620 04:35:35.779020 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.779022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.779064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.779130 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:35.779391 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.779998 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.781390 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.781614 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.781699 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.782029 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.782092 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.782162 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.782347 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.782401 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:35:35.782570 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.778244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37153 (local address 127.0.98.2:54973)
0620 04:35:35.778645 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.778733 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.778872 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.779161 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.779164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.779173 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:35.779286 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.779293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.780113 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.780117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.781241 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.781252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.781734 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.781737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.781834 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.781889 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.782183 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.782210 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.782320 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.782410 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":95,"threads_started":1}
I20260620 04:35:35.789500 1942 raft_consensus.cc:1261] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Refusing update from remote peer 841e6124331140eea5e45ac9e88cabf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.789814 1664 raft_consensus.cc:1261] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Refusing update from remote peer 841e6124331140eea5e45ac9e88cabf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:35.789901 2033 consensus_queue.cc:1035] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d67bfc9d8149529c862cae7ec1f97d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.790288 2033 consensus_queue.cc:1035] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d45c838a114e72a72f1aee8086ec78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:35.796317 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298785836190392320 to 7298785835380957184. Current Snapshot: MvccSnapshot[applied={T|T < 7298785836190392320}]
I20260620 04:35:35.812762 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.809056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33109 (local address 127.0.98.3:48385)
0620 04:35:35.809366 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:35.809378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:35.809498 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:35.809756 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:35.809759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:35.809771 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:35.809900 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.809907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.810865 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.810870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.811495 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:35.811502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.811944 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.811947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:35.812061 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.812099 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.812431 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.812438 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:35:35.812514 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:35:35.812590 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":123,"threads_started":1}
I20260620 04:35:35.812923 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.809121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48385 (local address 127.0.98.2:33109)
0620 04:35:35.809246 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.809252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.809475 (+ 223us) server_negotiation.cc:407] Connection header received
0620 04:35:35.809601 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.809604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.809655 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.809736 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:35.810038 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.810756 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.811609 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.811840 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.811947 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.812247 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:35.812323 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:35.812411 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.812737 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.812789 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:35:35.836256 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:35.836418 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:35.838450 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 691f21cc037e4072bc7555e9b27f51f7: Sending DeleteTablet for 3 replicas of tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61
I20260620 04:35:35.838946 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33858
I20260620 04:35:35.839119 2036 tablet_replica.cc:330] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: stopping tablet replica
I20260620 04:35:35.839349 2036 raft_consensus.cc:2229] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:35.839485 2036 raft_consensus.cc:2258] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:35.839833 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59432
I20260620 04:35:35.839995 2037 tablet_replica.cc:330] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: stopping tablet replica
I20260620 04:35:35.840095 2037 raft_consensus.cc:2229] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:35.840310 2037 raft_consensus.cc:2258] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:35.840969 2036 ts_tablet_manager.cc:1905] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.843924 2036 ts_tablet_manager.cc:1918] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:35.843993 2036 log.cc:1196] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/8739eb1fdbfc48ad9cd6b1c373fa2f61
I20260620 04:35:35.844494 2036 ts_tablet_manager.cc:1939] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 93d45c838a114e72a72f1aee8086ec78: Deleting consensus metadata
I20260620 04:35:35.844781 2037 ts_tablet_manager.cc:1905] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.845185 1491 catalog_manager.cc:4841] TS 93d45c838a114e72a72f1aee8086ec78 (127.0.98.1:42663): tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (table test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]) successfully deleted
I20260620 04:35:35.846904 2037 ts_tablet_manager.cc:1918] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:35.846998 2037 log.cc:1196] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/8739eb1fdbfc48ad9cd6b1c373fa2f61
I20260620 04:35:35.847219 2037 ts_tablet_manager.cc:1939] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P 841e6124331140eea5e45ac9e88cabf8: Deleting consensus metadata
I20260620 04:35:35.847796 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47360
I20260620 04:35:35.847833 1494 catalog_manager.cc:4841] TS 841e6124331140eea5e45ac9e88cabf8 (127.0.98.2:33109): tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (table test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]) successfully deleted
I20260620 04:35:35.847999 2038 tablet_replica.cc:330] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: stopping tablet replica
I20260620 04:35:35.848129 2038 raft_consensus.cc:2229] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:35.848270 2038 raft_consensus.cc:2258] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:35.848973 2038 ts_tablet_manager.cc:1905] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:35.852046 2038 ts_tablet_manager.cc:1918] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:35.852131 2038 log.cc:1196] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/8739eb1fdbfc48ad9cd6b1c373fa2f61
I20260620 04:35:35.852425 2038 ts_tablet_manager.cc:1939] T 8739eb1fdbfc48ad9cd6b1c373fa2f61 P e5d67bfc9d8149529c862cae7ec1f97d: Deleting consensus metadata
I20260620 04:35:35.853374 1494 catalog_manager.cc:4841] TS e5d67bfc9d8149529c862cae7ec1f97d (127.0.98.3:37153): tablet 8739eb1fdbfc48ad9cd6b1c373fa2f61 (table test-table [id=c4a10afa5f1c4cec8e07851fec8d9592]) successfully deleted
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930135, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:35.859441 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.848725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.0.98.1:42663)
0620 04:35:35.848857 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.848862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.849769 (+ 907us) server_negotiation.cc:407] Connection header received
0620 04:35:35.849931 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.849934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.849980 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.850076 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.850775 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.851285 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.852064 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.852409 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.852502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.855500 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.855511 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.855525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.855551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.857548 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.858074 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.858079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.858081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.858180 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.858463 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.858466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.858468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.858745 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.859052 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.859171 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.859296 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 04:35:35.859875 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37208
I20260620 04:35:35.873411 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:35.864583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.0.98.62:35975)
0620 04:35:35.864704 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:35.864709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:35.864812 (+ 103us) server_negotiation.cc:407] Connection header received
0620 04:35:35.864940 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:35.864943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:35.864997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:35.865084 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:35.865813 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.866644 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.867767 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:35.868143 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:35.868315 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:35.869834 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:35.869839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:35.869842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:35.869869 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:35.871583 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.872076 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.872079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.872081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.872165 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:35.872508 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:35.872512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:35.872514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:35.872751 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:35.872888 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:35.873123 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 04:35:35.873249 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:35:35.873766 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42396
I20260620 04:35:35.875008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1579
I20260620 04:35:35.883605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1719
I20260620 04:35:35.892282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1859
I20260620 04:35:35.899281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 1475
2026-06-20T04:35:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3112 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[2077](info): set up 2 sockets
Jun 20 04:35:35 dist-test-slave-zmf4 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260620 04:35:36.960520 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:37.080579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:37.185948 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:37.186336 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:37.186419 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:37.190371 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:37.190456 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:35:37.190488 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:37.190536 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:37.190572 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:37.190605 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:37.195901 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39289
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:37.196910 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:37.197923 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:37.206072 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.206039 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.206121 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:37.206123 2090 server_base.cc:1034] running on GCE node
I20260620 04:35:37.206921 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:37.207952 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:37.209156 2090 hybrid_clock.cc:648] HybridClock initialized: now 1781930137209152 us; error 55 us; skew 500 ppm
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:37.211874 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:37.212895 2090 webserver.cc:460] Webserver started at http://127.0.98.62:37531/ using document root <none> and password file <none>
I20260620 04:35:37.213439 2090 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:37.213608 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:37.213841 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:37.215818 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "c43a024b77ab4ca68f40272498b3865f"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "122524d2c88eff85685e88b87a8bb450"
server_key_iv: "5b61cb4ff2542d649ab41442d170ea89"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.216398 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c43a024b77ab4ca68f40272498b3865f"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "122524d2c88eff85685e88b87a8bb450"
server_key_iv: "5b61cb4ff2542d649ab41442d170ea89"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.219930 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:37.222391 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.223701 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:37.223850 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "c43a024b77ab4ca68f40272498b3865f"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "122524d2c88eff85685e88b87a8bb450"
server_key_iv: "5b61cb4ff2542d649ab41442d170ea89"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.223956 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:37.242671 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:37.245518 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:37.245708 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:37.253831 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39289
I20260620 04:35:37.253865 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39289 every 8 connection(s)
I20260620 04:35:37.254828 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:37.256017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2090
I20260620 04:35:37.256131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:37.256503 392 external_mini_cluster.cc:1428] Setting key 380f0ef8e2a4d5af4274a29250a19e7a
I20260620 04:35:37.258302 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:37.263823 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Bootstrap starting.
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:37.266239 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:37.267089 2168 log.cc:824] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:37.269168 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: No bootstrap required, opened a new log
I20260620 04:35:37.270913 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.258011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.98.62:39289)
0620 04:35:37.258527 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.258542 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.258577 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:35:37.259190 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.259205 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.259512 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.259853 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.260624 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.261584 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.262428 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.262890 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.263050 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.265547 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.265552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.265563 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.265589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.268001 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.268509 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.268514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.268518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.268624 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.268911 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.268914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.268916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.269309 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.269657 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.269977 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.270246 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":145,"threads_started":1}
I20260620 04:35:37.272619 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } }
I20260620 04:35:37.272881 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:37.272964 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43a024b77ab4ca68f40272498b3865f, State: Initialized, Role: FOLLOWER
I20260620 04:35:37.273398 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } }
I20260620 04:35:37.273643 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:37.273741 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:37.273840 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:37.274670 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } }
I20260620 04:35:37.274989 2168 leader_election.cc:302] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43a024b77ab4ca68f40272498b3865f; no voters:
I20260620 04:35:37.275308 2168 leader_election.cc:288] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:37.275400 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:37.275632 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [term 1 LEADER]: Becoming Leader. State: Replica: c43a024b77ab4ca68f40272498b3865f, State: Running, Role: LEADER
I20260620 04:35:37.275914 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } }
I20260620 04:35:37.276482 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:37.277531 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43a024b77ab4ca68f40272498b3865f. Latest consensus state: current_term: 1 leader_uuid: "c43a024b77ab4ca68f40272498b3865f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } } }
I20260620 04:35:37.277676 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:37.278015 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43a024b77ab4ca68f40272498b3865f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43a024b77ab4ca68f40272498b3865f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39289 } } }
I20260620 04:35:37.278108 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:37.278071 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:37.281437 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:37.287050 2182 catalog_manager.cc:1347] Generated new cluster ID: 6cbbdbbe8e9448deae129f5bbef14a8f
I20260620 04:35:37.287123 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:37.318681 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:37.319630 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:37.324924 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Generated new TSK 0
I20260620 04:35:37.325543 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:37.359129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39289
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:37.470475 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:37.470738 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:37.470793 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:37.475255 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:37.475310 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:37.475354 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:35:37.475416 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:37.475486 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:37.481544 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:37.482614 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:37.483781 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:37.493633 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.493630 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.493819 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:37.494035 2194 server_base.cc:1034] running on GCE node
I20260620 04:35:37.494462 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:37.495174 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:37.496358 2194 hybrid_clock.cc:648] HybridClock initialized: now 1781930137496353 us; error 32 us; skew 500 ppm
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:37.499257 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:37.500174 2194 webserver.cc:460] Webserver started at http://127.0.98.1:38761/ using document root <none> and password file <none>
I20260620 04:35:37.500921 2194 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:37.501024 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:37.501255 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:37.503213 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "06224bd9bd194073adc09fda4361ee51"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "1fd0d991233fda334b49b1d6958b88e3"
server_key_iv: "81152beafbf646e0403693a17852f1ea"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.503773 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06224bd9bd194073adc09fda4361ee51"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "1fd0d991233fda334b49b1d6958b88e3"
server_key_iv: "81152beafbf646e0403693a17852f1ea"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.507378 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:35:37.509843 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.511185 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:37.511332 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "06224bd9bd194073adc09fda4361ee51"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "1fd0d991233fda334b49b1d6958b88e3"
server_key_iv: "81152beafbf646e0403693a17852f1ea"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.511448 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:37.533429 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:37.536667 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:37.536891 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:37.538434 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:37.538506 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.538579 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:37.538630 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:37.550112 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46605
I20260620 04:35:37.550297 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46605 every 8 connection(s)
I20260620 04:35:37.551473 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:37.553790 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.539247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33739 (local address 127.0.98.62:39289)
0620 04:35:37.539417 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.539424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.540348 (+ 924us) server_negotiation.cc:407] Connection header received
0620 04:35:37.541746 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.541757 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.541877 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.541970 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.543915 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.544605 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.546485 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.546861 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.546946 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.549619 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.549622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.549625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.549658 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.551385 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.551959 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.551964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.551966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.552054 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.552465 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.552468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.552470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.552691 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.552778 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.553553 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.553671 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:35:37.554756 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.539652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.1:33739)
0620 04:35:37.540145 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.540228 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.541422 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.542516 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.542529 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.543007 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.543740 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.543763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.544736 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.544741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.546347 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.546359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.546988 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.546992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.547194 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.547948 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.547963 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.549440 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.551525 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.551530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.551544 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.551854 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.552168 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.552171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.552173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.552354 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.552902 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:37.552907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:37.553144 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 04:35:37.553394 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 04:35:37.553871 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260620 04:35:37.555375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2194
I20260620 04:35:37.555496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:37.555823 392 external_mini_cluster.cc:1428] Setting key 35faf3bb0915f01961639bfcbfa1a2c9
I20260620 04:35:37.555989 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39289
I20260620 04:35:37.558238 2325 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:37.558732 2325 heartbeater.cc:511] Master 127.0.98.62:39289 requested a full tablet report, sending...
I20260620 04:35:37.560329 2121 ts_manager.cc:194] Registered new tserver with Master: 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605)
I20260620 04:35:37.561939 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33739
I20260620 04:35:37.565620 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.558425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34533 (local address 127.0.98.62:39289)
0620 04:35:37.558551 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.558557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.558886 (+ 329us) server_negotiation.cc:407] Connection header received
0620 04:35:37.559013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.559016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.559052 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.559253 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.559720 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.560271 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.561104 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.561335 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.561440 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.562598 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.562601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.562604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.562627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.564024 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.564481 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.564486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.564488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.564563 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.564856 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.564859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.564861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.565018 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.565120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.565378 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.565479 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:35:37.565626 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.558343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.1:34533)
0620 04:35:37.558816 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.558831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.558920 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.559267 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.559270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.559481 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.559600 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.559605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.560386 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.560389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.560985 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.560994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.561443 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.561447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.561538 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.562084 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.562098 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.562428 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.564137 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.564142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.564144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.564383 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.564666 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.564669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.564671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.564761 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.565120 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:37.565124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:37.565213 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 04:35:37.565316 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:35:37.565446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:37.584417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39289
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:37.697793 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:37.698092 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:37.698148 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:37.702672 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:37.702745 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:37.702790 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:35:37.702930 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:37.702981 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:37.708525 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:37.709555 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:37.710546 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:37.720506 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.720522 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.720544 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:37.720914 2334 server_base.cc:1034] running on GCE node
I20260620 04:35:37.721474 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:37.722237 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:37.723418 2334 hybrid_clock.cc:648] HybridClock initialized: now 1781930137723405 us; error 26 us; skew 500 ppm
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:37.726410 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:37.727331 2334 webserver.cc:460] Webserver started at http://127.0.98.2:38743/ using document root <none> and password file <none>
I20260620 04:35:37.727895 2334 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:37.727957 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:37.728178 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:37.730293 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "92484a410dd744cebb7b25b76406a899"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "f5079c6c034259dc9e3731872af60ef9"
server_key_iv: "493cb0ea63f46bf93d488f113885613e"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.730810 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92484a410dd744cebb7b25b76406a899"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "f5079c6c034259dc9e3731872af60ef9"
server_key_iv: "493cb0ea63f46bf93d488f113885613e"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.734754 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:35:37.737329 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.738564 2334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:37.738726 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "92484a410dd744cebb7b25b76406a899"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "f5079c6c034259dc9e3731872af60ef9"
server_key_iv: "493cb0ea63f46bf93d488f113885613e"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.738842 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:37.752014 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:37.755167 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:37.755373 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:37.757141 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:37.757222 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.757315 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:37.757342 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:37.768131 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34429
I20260620 04:35:37.768224 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34429 every 8 connection(s)
I20260620 04:35:37.769060 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:37.770473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2334
I20260620 04:35:37.770555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:37.770768 392 external_mini_cluster.cc:1428] Setting key df2db646296873f6b41d1bad00dc24d3
I20260620 04:35:37.771198 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.757864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41215 (local address 127.0.98.62:39289)
0620 04:35:37.757979 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.757983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.759000 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 04:35:37.759885 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.759889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.759947 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.760056 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.761291 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.761925 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.763162 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.763571 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.763713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.767262 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.767268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.767276 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.767301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.768906 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.769469 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.769474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.769476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.769568 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.769937 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.769939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.769940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.770119 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.770200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.770894 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.771020 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 04:35:37.772096 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.758397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.2:41215)
0620 04:35:37.758859 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.758898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.759690 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.760259 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.760267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.760696 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.761139 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.761152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.762069 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.762072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.763060 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.763070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.763763 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.763767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.763967 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.764616 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.764640 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.767076 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.769030 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.769035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.769048 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.769364 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.769684 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.769687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.769689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.769846 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.770312 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:37.770320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:37.770633 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 04:35:37.770851 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 04:35:37.771154 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260620 04:35:37.773406 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39289
I20260620 04:35:37.773545 2465 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:37.774082 2465 heartbeater.cc:511] Master 127.0.98.62:39289 requested a full tablet report, sending...
I20260620 04:35:37.775215 2122 ts_manager.cc:194] Registered new tserver with Master: 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:37.775789 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41215
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:37.788724 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.780514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.2:53187)
0620 04:35:37.780929 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.780946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.781064 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.781328 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.781331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.781498 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.781592 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.781597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.782458 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.782461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.783020 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.783028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.783684 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.783686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.783780 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.784444 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.784464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.784948 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.787007 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.787010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.787011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.787265 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.787641 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.787643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.787644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.787722 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.788285 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:37.788288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:37.788368 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 04:35:37.788462 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:35:37.788574 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260620 04:35:37.788908 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.780630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53187 (local address 127.0.98.62:39289)
0620 04:35:37.780765 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.780769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.781049 (+ 280us) server_negotiation.cc:407] Connection header received
0620 04:35:37.781161 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.781164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.781201 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.781286 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.781709 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.782309 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.783139 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.783563 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.783700 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.785124 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.785131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.785136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.785167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.786889 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.787436 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.787441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.787443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.787539 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.787820 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.787824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.787826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.788108 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.788250 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.788552 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.788686 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:37.799630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39289
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:37.914126 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:37.914470 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:37.914563 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:37.919008 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:37.919106 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:37.919173 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:35:37.919219 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:35:37.919291 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:37.924582 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:37.925586 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:37.926561 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:37.935874 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.935892 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:37.935882 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:37.936301 2474 server_base.cc:1034] running on GCE node
I20260620 04:35:37.936690 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:37.937263 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:37.938431 2474 hybrid_clock.cc:648] HybridClock initialized: now 1781930137938395 us; error 47 us; skew 500 ppm
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:37.941143 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:37.942063 2474 webserver.cc:460] Webserver started at http://127.0.98.3:38275/ using document root <none> and password file <none>
I20260620 04:35:37.942633 2474 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:37.942735 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:37.943010 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:37.945041 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "c689f3cd76ac4739b5c07cfaa686f77a"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "19820a53336bd7a9fca56e4b1237a094"
server_key_iv: "1e7a17d7fbf8a5a19be8c7e47ba4dc5a"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.945588 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c689f3cd76ac4739b5c07cfaa686f77a"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "19820a53336bd7a9fca56e4b1237a094"
server_key_iv: "1e7a17d7fbf8a5a19be8c7e47ba4dc5a"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.949116 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:37.951465 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.952865 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:37.953003 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "c689f3cd76ac4739b5c07cfaa686f77a"
format_stamp: "Formatted at 2026-06-20 04:35:37 on dist-test-slave-zmf4"
server_key: "19820a53336bd7a9fca56e4b1237a094"
server_key_iv: "1e7a17d7fbf8a5a19be8c7e47ba4dc5a"
server_key_version: "encryptionkey@0"
I20260620 04:35:37.953120 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:37.968014 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:37.970840 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:37.971036 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:37.972446 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:37.972492 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:37.972555 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:37.972586 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:35:37 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:37.983574 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39005
I20260620 04:35:37.983695 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39005 every 8 connection(s)
I20260620 04:35:37.984887 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:37.985505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2474
I20260620 04:35:37.985625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:37.986017 392 external_mini_cluster.cc:1428] Setting key 33a820791941fd83d68f4461381d8abe
I20260620 04:35:37.988035 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.973875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.3:49395)
0620 04:35:37.974350 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.974398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.975693 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.976374 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.976383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.976790 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.977464 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.977481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.978632 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.978637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.979721 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.979730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.980270 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.980274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.980464 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.981128 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.981145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.982719 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.984854 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.984860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.984872 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.985257 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.985591 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.985595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.985597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.985767 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.986215 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:37.986221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:37.986521 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 04:35:37.986717 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 04:35:37.987064 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260620 04:35:37.988672 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.973333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49395 (local address 127.0.98.62:39289)
0620 04:35:37.973567 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.973572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.974514 (+ 942us) server_negotiation.cc:407] Connection header received
0620 04:35:37.975973 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.975980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.976051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.976165 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.977606 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.978508 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.979839 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.980074 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.980196 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.982892 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.982899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.982903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.982933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.984721 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.985393 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.985396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.985398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.985487 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.985870 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.985876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.985878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.986077 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:37.986194 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:37.988432 (+ 2238us) server_negotiation.cc:299] Negotiation successful
0620 04:35:37.988565 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 04:35:37.989665 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39289
I20260620 04:35:37.989892 2605 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:37.990437 2605 heartbeater.cc:511] Master 127.0.98.62:39289 requested a full tablet report, sending...
I20260620 04:35:37.991554 2121 ts_manager.cc:194] Registered new tserver with Master: c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005)
I20260620 04:35:37.992142 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49395
I20260620 04:35:37.999725 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:35:38.000734 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.992898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42067 (local address 127.0.98.62:39289)
0620 04:35:37.993014 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:37.993018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:37.993196 (+ 178us) server_negotiation.cc:407] Connection header received
0620 04:35:37.993284 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:37.993287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:37.993326 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:37.993400 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:37.993934 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.994467 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.995235 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.995474 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.995559 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.996915 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:37.996920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:37.996922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:37.996969 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:37.998912 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.999381 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.999387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.999389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.999475 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:37.999782 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:37.999788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:37.999792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:37.999986 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:38.000105 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.000448 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.000563 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:35:38.000744 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:37.992793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.98.3:42067)
0620 04:35:37.993085 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:37.993101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:37.993199 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:37.993436 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:37.993439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:37.993657 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:37.993773 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.993780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.994586 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.994589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.995119 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:37.995126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:37.995617 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:37.995620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:37.995700 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:37.996292 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:37.996308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:37.996671 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:37.999038 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.999043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.999046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.999271 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:37.999576 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:37.999582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:37.999584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:37.999673 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:38.000093 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:38.000096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:38.000255 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.000389 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.000540 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
Jun 20 04:35:38 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:35:38 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:38.029567 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.018208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.98.62:39289)
0620 04:35:38.018338 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.018342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.018354 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:35:38.018391 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.018393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.018428 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.018499 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:38.019224 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.019732 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.020902 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.021526 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.021640 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.025282 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:38.025292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:38.025311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:38.025344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:38.027707 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:38.028246 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:38.028249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:38.028251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:38.028330 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:38.028720 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:38.028722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:38.028723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:38.028950 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:38.029048 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.029327 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.029422 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:35:38.036090 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:38.039917 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:38.058377 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.052237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.0.1:41042)
0620 04:35:38.052713 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.052728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.052831 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.053311 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.053314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.053332 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.053446 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.053452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.054872 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.054875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.056321 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.056331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.057809 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.057813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.057940 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.057981 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.058091 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.058210 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260620 04:35:38.059186 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.052386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.3:39005)
0620 04:35:38.052723 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.052739 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.052804 (+ 65us) server_negotiation.cc:407] Connection header received
0620 04:35:38.052957 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.052963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.053156 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.053308 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.053696 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.054756 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.056450 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.057455 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.057560 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.057742 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.058919 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.059038 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260620 04:35:38.059428 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.051069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.98.2:34429)
0620 04:35:38.051989 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.052010 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.052038 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:35:38.052124 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.052131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.052933 (+ 802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.053084 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.053365 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.054910 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.056723 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.058264 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.058354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.058599 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.059105 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.059223 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":98,"threads_started":1}
I20260620 04:35:38.059860 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.050945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34429 (local address 127.0.0.1:37226)
0620 04:35:38.051300 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.051352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.051482 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.053091 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.053097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.053119 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.053271 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.053278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.055021 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.055024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.056562 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.056574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.058710 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.058716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.058913 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.058949 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.059581 (+ 632us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.059676 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":29,"thread_start_us":83,"threads_started":1}
I20260620 04:35:38.060912 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.051081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46605 (local address 127.0.0.1:43266)
0620 04:35:38.052468 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.052489 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.052595 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.055626 (+ 3031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.055629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.055645 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.055729 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.055734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.057346 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.057349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.058914 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.058926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.060512 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.060516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.060596 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.060614 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.060711 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.060781 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"mutex_wait_us":46,"thread_start_us":89,"threads_started":1}
I20260620 04:35:38.060992 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.051768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.98.1:46605)
0620 04:35:38.055057 (+ 3289us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.055077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.055099 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:35:38.055166 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.055171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.055350 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.055494 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.055845 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.057163 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.059054 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.060214 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.060317 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.060450 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.060744 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.060841 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":59,"threads_started":1}
I20260620 04:35:38.062556 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 (DEFAULT_TABLE table=test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.063830 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5046857c064fb1b55d71b2fa188f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.067034 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 (DEFAULT_TABLE table=test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.067029 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 (DEFAULT_TABLE table=test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.068096 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5046857c064fb1b55d71b2fa188f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.068096 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5046857c064fb1b55d71b2fa188f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.073112 2628 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Bootstrap starting.
I20260620 04:35:38.075706 2628 tablet_bootstrap.cc:654] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.076489 2630 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Bootstrap starting.
I20260620 04:35:38.076607 2628 log.cc:824] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:38.077798 2629 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Bootstrap starting.
I20260620 04:35:38.079000 2630 tablet_bootstrap.cc:654] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.079818 2630 log.cc:824] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:38.079978 2629 tablet_bootstrap.cc:654] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.080838 2629 log.cc:824] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:38.081970 2630 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: No bootstrap required, opened a new log
I20260620 04:35:38.082152 2630 ts_tablet_manager.cc:1397] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:35:38.083667 2629 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: No bootstrap required, opened a new log
I20260620 04:35:38.083914 2629 ts_tablet_manager.cc:1397] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 04:35:38.085490 2628 tablet_bootstrap.cc:492] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: No bootstrap required, opened a new log
I20260620 04:35:38.085690 2628 ts_tablet_manager.cc:1397] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260620 04:35:38.088627 2629 raft_consensus.cc:348] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.089018 2629 raft_consensus.cc:374] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.089112 2629 raft_consensus.cc:729] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.089638 2629 consensus_queue.cc:260] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.089676 2628 raft_consensus.cc:348] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.089915 2628 raft_consensus.cc:374] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.089977 2628 raft_consensus.cc:729] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.090373 2628 consensus_queue.cc:260] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.091121 2605 heartbeater.cc:503] Master 127.0.98.62:39289 was elected leader, sending a full tablet report...
I20260620 04:35:38.092025 2629 ts_tablet_manager.cc:1428] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260620 04:35:38.092216 2325 heartbeater.cc:503] Master 127.0.98.62:39289 was elected leader, sending a full tablet report...
I20260620 04:35:38.093359 2628 ts_tablet_manager.cc:1428] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 04:35:38.095286 2630 raft_consensus.cc:348] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.095558 2630 raft_consensus.cc:374] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.095609 2630 raft_consensus.cc:729] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.096055 2630 consensus_queue.cc:260] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.096776 2465 heartbeater.cc:503] Master 127.0.98.62:39289 was elected leader, sending a full tablet report...
I20260620 04:35:38.097373 2630 ts_tablet_manager.cc:1428] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260620 04:35:38.128252 2635 raft_consensus.cc:482] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:38.128527 2635 raft_consensus.cc:504] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.129963 2635 leader_election.cc:288] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429), c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005)
I20260620 04:35:38.135325 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.130418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53327 (local address 127.0.98.2:34429)
0620 04:35:38.130594 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.130599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.130749 (+ 150us) server_negotiation.cc:407] Connection header received
0620 04:35:38.130961 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.130965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.131023 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.131076 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.131594 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.132351 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.133936 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.134539 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.134643 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.134715 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.135155 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.135226 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:35:38.135361 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.130263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34429 (local address 127.0.98.1:53327)
0620 04:35:38.130658 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.130670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.130798 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.131236 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.131238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.131272 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.131434 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.131440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.132474 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.132478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.133801 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.133811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.134675 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.134679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.134862 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.134896 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.135112 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.135191 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":29,"thread_start_us":130,"threads_started":1}
I20260620 04:35:38.136098 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.130820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36297 (local address 127.0.98.3:39005)
0620 04:35:38.130952 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.130956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.131062 (+ 106us) server_negotiation.cc:407] Connection header received
0620 04:35:38.131174 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.131176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.131214 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.131276 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.132347 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.133208 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.134661 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.135487 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.135595 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.135666 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.135929 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.136013 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:38.136106 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.130700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.1:36297)
0620 04:35:38.130991 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.131004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.131094 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.131299 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.131302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.131315 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.131434 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.131440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.133343 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.133424 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.134478 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.134489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.135722 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.135726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.135811 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.135828 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.135887 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.135935 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260620 04:35:38.136282 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5046857c064fb1b55d71b2fa188f92" candidate_uuid: "06224bd9bd194073adc09fda4361ee51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899" is_pre_election: true
I20260620 04:35:38.136626 2417 raft_consensus.cc:2454] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06224bd9bd194073adc09fda4361ee51 in term 0.
I20260620 04:35:38.136689 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5046857c064fb1b55d71b2fa188f92" candidate_uuid: "06224bd9bd194073adc09fda4361ee51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" is_pre_election: true
I20260620 04:35:38.136940 2559 raft_consensus.cc:2454] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06224bd9bd194073adc09fda4361ee51 in term 0.
I20260620 04:35:38.137163 2211 leader_election.cc:302] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06224bd9bd194073adc09fda4361ee51, 92484a410dd744cebb7b25b76406a899; no voters:
I20260620 04:35:38.137462 2635 raft_consensus.cc:2781] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:38.137534 2635 raft_consensus.cc:482] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:38.137595 2635 raft_consensus.cc:3037] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.138605 2635 raft_consensus.cc:504] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.138942 2635 leader_election.cc:288] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [CANDIDATE]: Term 1 election: Requested vote from peers 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429), c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005)
I20260620 04:35:38.139405 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5046857c064fb1b55d71b2fa188f92" candidate_uuid: "06224bd9bd194073adc09fda4361ee51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899"
I20260620 04:35:38.139475 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5046857c064fb1b55d71b2fa188f92" candidate_uuid: "06224bd9bd194073adc09fda4361ee51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c689f3cd76ac4739b5c07cfaa686f77a"
I20260620 04:35:38.139542 2417 raft_consensus.cc:3037] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.139635 2559 raft_consensus.cc:3037] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.140712 2559 raft_consensus.cc:2454] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06224bd9bd194073adc09fda4361ee51 in term 1.
I20260620 04:35:38.140776 2417 raft_consensus.cc:2454] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06224bd9bd194073adc09fda4361ee51 in term 1.
I20260620 04:35:38.141063 2211 leader_election.cc:302] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06224bd9bd194073adc09fda4361ee51, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:38.141220 2635 raft_consensus.cc:2781] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:38.141453 2635 raft_consensus.cc:686] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 1 LEADER]: Becoming Leader. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Running, Role: LEADER
I20260620 04:35:38.141795 2635 consensus_queue.cc:237] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.145089 2122 catalog_manager.cc:5495] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06224bd9bd194073adc09fda4361ee51 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "06224bd9bd194073adc09fda4361ee51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:35:38 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:38.184260 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.174322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.0.98.62:39289)
0620 04:35:38.174570 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.174575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.175141 (+ 566us) server_negotiation.cc:407] Connection header received
0620 04:35:38.175185 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.175187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.175227 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.175307 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:38.176267 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.177043 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.178127 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.178332 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.178415 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.180958 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:38.180961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:38.180963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:38.180983 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:38.182419 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:38.182926 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:38.182929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:38.182930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:38.183006 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:38.183376 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:38.183379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:38.183380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:38.183565 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:38.183645 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.183983 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.184093 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 04:35:38.184425 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.174258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39289 (local address 127.0.0.1:50748)
0620 04:35:38.174744 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.174758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.174995 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.175432 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.175434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.175921 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:38.176048 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.176056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.177197 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.177199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.177980 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.177989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.178602 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.178605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.178670 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.179220 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:38.179231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:38.180728 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:38.182544 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:38.182551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:38.182556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:38.182789 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:38.183136 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:38.183139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:38.183140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:38.183238 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:38.183695 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:38.183701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:38.183805 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.183908 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.184025 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":230,"threads_started":1}
I20260620 04:35:38.189419 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:35:38.190101 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:38.195494 2259 tablet_service.cc:1467] Processing CreateTablet for tablet ca1be9fde16246a48eda9de36d2fb292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39c8bde66b654817875b60d5eaa924dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:38.196175 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1be9fde16246a48eda9de36d2fb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.195888 2399 tablet_service.cc:1467] Processing CreateTablet for tablet ca1be9fde16246a48eda9de36d2fb292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39c8bde66b654817875b60d5eaa924dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:38.195894 2539 tablet_service.cc:1467] Processing CreateTablet for tablet ca1be9fde16246a48eda9de36d2fb292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39c8bde66b654817875b60d5eaa924dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:35:38.196583 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1be9fde16246a48eda9de36d2fb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.196592 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1be9fde16246a48eda9de36d2fb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.198734 2629 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: Bootstrap starting.
I20260620 04:35:38.199404 2630 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: Bootstrap starting.
I20260620 04:35:38.199456 2628 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: Bootstrap starting.
I20260620 04:35:38.199738 2629 tablet_bootstrap.cc:654] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.200505 2628 tablet_bootstrap.cc:654] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.200505 2630 tablet_bootstrap.cc:654] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.201308 2629 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: No bootstrap required, opened a new log
I20260620 04:35:38.201402 2629 ts_tablet_manager.cc:1397] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:38.201790 2628 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: No bootstrap required, opened a new log
I20260620 04:35:38.201869 2628 ts_tablet_manager.cc:1397] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:38.201908 2629 raft_consensus.cc:348] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.202020 2630 tablet_bootstrap.cc:492] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: No bootstrap required, opened a new log
I20260620 04:35:38.202020 2629 raft_consensus.cc:374] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.202056 2629 raft_consensus.cc:729] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.202091 2630 ts_tablet_manager.cc:1397] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:38.202203 2629 consensus_queue.cc:260] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.202217 2628 raft_consensus.cc:348] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.202316 2628 raft_consensus.cc:374] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.202365 2628 raft_consensus.cc:729] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.202467 2629 ts_tablet_manager.cc:1428] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:35:38.202479 2628 consensus_queue.cc:260] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.202473 2635 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.202517 2630 raft_consensus.cc:348] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.202625 2635 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.202636 2630 raft_consensus.cc:374] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.202680 2630 raft_consensus.cc:729] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.202732 2628 ts_tablet_manager.cc:1428] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:38.202791 2630 consensus_queue.cc:260] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.203069 2630 ts_tablet_manager.cc:1428] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:35:38.203034 2636 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.203168 2636 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.203931 2634 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.204061 2634 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.209834 2634 raft_consensus.cc:482] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:38.209934 2634 raft_consensus.cc:504] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.210932 2634 leader_election.cc:288] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:38.214775 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.211171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34429 (local address 127.0.98.3:55257)
0620 04:35:38.211365 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.211383 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.211466 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.211771 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.211774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.211802 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.211941 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.211948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.212902 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.212908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.213669 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.213677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.214480 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.214482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.214553 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.214569 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.214628 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.214681 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 04:35:38.214854 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.211219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55257 (local address 127.0.98.2:34429)
0620 04:35:38.211422 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.211427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.211441 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:35:38.211578 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.211583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.211638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.211722 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.212075 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.212798 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.213786 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.214251 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.214374 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.214431 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.214648 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.214739 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 04:35:38.214934 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.211219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54461 (local address 127.0.98.1:46605)
0620 04:35:38.211326 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.211330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.211725 (+ 395us) server_negotiation.cc:407] Connection header received
0620 04:35:38.211779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.211782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.211829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.211937 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.212224 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.212861 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.213866 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.214396 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.214497 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.214553 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.214759 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.214823 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:35:38.215204 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1be9fde16246a48eda9de36d2fb292" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899" is_pre_election: true
I20260620 04:35:38.215261 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.211171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46605 (local address 127.0.98.3:54461)
0620 04:35:38.211562 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.211586 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.211693 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.211950 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.211953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.211969 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.212101 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.212111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.212967 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.212972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.213745 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.213753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.214510 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.214514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.214624 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.214646 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.215047 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.215123 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":48,"thread_start_us":121,"threads_started":1}
I20260620 04:35:38.215345 2417 raft_consensus.cc:2454] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:38.215677 2491 leader_election.cc:302] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:38.215931 2634 raft_consensus.cc:2781] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:38.215920 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1be9fde16246a48eda9de36d2fb292" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51" is_pre_election: true
I20260620 04:35:38.215982 2634 raft_consensus.cc:482] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:38.216022 2634 raft_consensus.cc:3037] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.216105 2279 raft_consensus.cc:2454] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:38.217309 2634 raft_consensus.cc:504] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.217661 2634 leader_election.cc:288] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Requested vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:38.217999 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1be9fde16246a48eda9de36d2fb292" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51"
I20260620 04:35:38.218011 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1be9fde16246a48eda9de36d2fb292" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899"
I20260620 04:35:38.218097 2279 raft_consensus.cc:3037] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.218132 2417 raft_consensus.cc:3037] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.218968 2417 raft_consensus.cc:2454] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:38.219020 2279 raft_consensus.cc:2454] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:38.219231 2491 leader_election.cc:302] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:38.219396 2634 raft_consensus.cc:2781] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:38.219611 2634 raft_consensus.cc:686] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 LEADER]: Becoming Leader. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Running, Role: LEADER
I20260620 04:35:38.219923 2634 consensus_queue.cc:237] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.222028 2651 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: TxnStatusTablet state changed. Reason: New leader c689f3cd76ac4739b5c07cfaa686f77a. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.222143 2651 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:38.222677 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:35:38.222858 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:35:38.223242 2122 catalog_manager.cc:5495] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a reported cstate change: term changed from 0 to 1, leader changed from <none> to c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: HEALTHY } } }
W20260620 04:35:38.237011 2606 tablet.cc:2369] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:38.240222 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.236236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.0.98.3:39005)
0620 04:35:38.236373 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.236378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.236393 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:35:38.236569 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.236579 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.236629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.236723 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.237007 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.237808 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.238676 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.239014 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.239116 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.239359 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.239832 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.239939 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.239996 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.240052 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:35:38.240162 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.236097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.0.1:41048)
0620 04:35:38.236290 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.236308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.236416 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.236744 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.236768 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.236775 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:38.236890 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.236897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.237967 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.237970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.238567 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.238574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.239118 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.239121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.239189 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.239230 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.239921 (+ 691us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.239931 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.239980 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.240051 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 04:35:38.245361 2279 raft_consensus.cc:1261] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.245501 2417 raft_consensus.cc:1261] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.246013 2651 consensus_queue.cc:1035] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.246295 2634 consensus_queue.cc:1035] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.249878 2635 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: TxnStatusTablet state changed. Reason: New leader c689f3cd76ac4739b5c07cfaa686f77a. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.250000 2635 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.252794 2636 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: TxnStatusTablet state changed. Reason: New leader c689f3cd76ac4739b5c07cfaa686f77a. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.252913 2636 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.253152 2635 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.253270 2635 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 06224bd9bd194073adc09fda4361ee51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.253931 2656 mvcc.cc:205] Tried to move back new op lower bound from 7298785846246371328 to 7298785846151430144. Current Snapshot: MvccSnapshot[applied={T|T < 7298785846246371328}]
I20260620 04:35:38.254386 2651 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.254491 2651 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:38.258239 2651 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.258350 2651 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P c689f3cd76ac4739b5c07cfaa686f77a: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:35:38.259896 2636 tablet_replica.cc:439] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260620 04:35:38.260002 2636 tablet_replica.cc:442] T ca1be9fde16246a48eda9de36d2fb292 P 92484a410dd744cebb7b25b76406a899: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:35:38.260344 2657 mvcc.cc:205] Tried to move back new op lower bound from 7298785846246371328 to 7298785846151430144. Current Snapshot: MvccSnapshot[applied={T|T < 7298785846246371328 or (T in {7298785846246371328})}]
I20260620 04:35:38.261765 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298785846246371328 to 7298785846151430144. Current Snapshot: MvccSnapshot[applied={T|T < 7298785846246371328}]
I20260620 04:35:38.269354 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.264889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.98.1:46605)
0620 04:35:38.264998 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.265005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.265020 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:35:38.265224 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.265227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.265288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.265385 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.265673 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.266446 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.267669 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.268016 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.268122 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.268445 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.268969 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.269085 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.269160 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.269220 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 04:35:38.270196 2466 tablet.cc:2369] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:38.281368 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.277670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42209 (local address 127.0.98.3:39005)
0620 04:35:38.277826 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.277831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.277961 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:35:38.278106 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.278109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.278162 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.278242 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.278476 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.279241 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.280134 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.280513 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.280627 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.280941 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.281019 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.281091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.281182 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.281231 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:35:38.281422 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.277511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.1:42209)
0620 04:35:38.277814 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.277882 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.278000 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.278256 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.278258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.278264 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:38.278379 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.278383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.279367 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.279370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.279992 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.280000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.280645 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.280648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.280744 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.280787 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.281115 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.281120 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.281170 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.281271 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260620 04:35:38.289763 2417 raft_consensus.cc:1261] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Refusing update from remote peer 06224bd9bd194073adc09fda4361ee51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.289991 2559 raft_consensus.cc:1261] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Refusing update from remote peer 06224bd9bd194073adc09fda4361ee51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.290218 2639 consensus_queue.cc:1035] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.290434 2635 consensus_queue.cc:1035] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.315169 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.310452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46605 (local address 127.0.98.3:44061)
0620 04:35:38.310616 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.310632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.310710 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.310945 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.310947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.310952 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:38.311045 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.311049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.312021 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.312024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.312729 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.312738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.314375 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.314378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.314470 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.314517 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.314916 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.314923 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.314990 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.315063 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 04:35:38.315222 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.310553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44061 (local address 127.0.98.1:46605)
0620 04:35:38.310665 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.310669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.310685 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:35:38.310801 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.310804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.310855 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.310932 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.311136 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.311923 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.313904 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.314256 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.314364 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.314710 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.314791 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.314871 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.315020 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.315094 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:35:38.342105 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:38.342298 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:38.344820 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Sending DeleteTablet for 3 replicas of tablet 0a5046857c064fb1b55d71b2fa188f92
I20260620 04:35:38.345517 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37226
I20260620 04:35:38.345992 2675 tablet_replica.cc:330] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: stopping tablet replica
I20260620 04:35:38.346287 2675 raft_consensus.cc:2229] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:38.346634 2675 raft_consensus.cc:2258] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:38.346947 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:38.347397 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:38.348165 2675 ts_tablet_manager.cc:1905] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:38.351647 2675 ts_tablet_manager.cc:1918] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:38.351755 2675 log.cc:1196] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/0a5046857c064fb1b55d71b2fa188f92
I20260620 04:35:38.352160 2675 ts_tablet_manager.cc:1939] T 0a5046857c064fb1b55d71b2fa188f92 P 92484a410dd744cebb7b25b76406a899: Deleting consensus metadata
I20260620 04:35:38.353484 2399 tablet_service.cc:1467] Processing CreateTablet for tablet acaac4b488064779bc6d5e4790c16c52 (DEFAULT_TABLE table=test-table [id=7df240935acc4374ae7eec8edd2801b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.353786 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaac4b488064779bc6d5e4790c16c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.355762 2630 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Bootstrap starting.
I20260620 04:35:38.355993 2107 catalog_manager.cc:4841] TS 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429): tablet 0a5046857c064fb1b55d71b2fa188f92 (table test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]) successfully deleted
I20260620 04:35:38.356741 2630 tablet_bootstrap.cc:654] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.356755 2538 tablet_service.cc:1467] Processing CreateTablet for tablet acaac4b488064779bc6d5e4790c16c52 (DEFAULT_TABLE table=test-table [id=7df240935acc4374ae7eec8edd2801b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.357041 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaac4b488064779bc6d5e4790c16c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.357961 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41042
I20260620 04:35:38.358114 2630 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: No bootstrap required, opened a new log
I20260620 04:35:38.358179 2630 ts_tablet_manager.cc:1397] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 04:35:38.358642 2630 raft_consensus.cc:348] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.358767 2630 raft_consensus.cc:374] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.358717 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5046857c064fb1b55d71b2fa188f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43266
I20260620 04:35:38.358798 2630 raft_consensus.cc:729] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.358913 2630 consensus_queue.cc:260] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.359191 2630 ts_tablet_manager.cc:1428] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:35:38.359791 2258 tablet_service.cc:1467] Processing CreateTablet for tablet acaac4b488064779bc6d5e4790c16c52 (DEFAULT_TABLE table=test-table [id=7df240935acc4374ae7eec8edd2801b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:38.360060 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaac4b488064779bc6d5e4790c16c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:38.360344 2677 tablet_replica.cc:330] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: stopping tablet replica
I20260620 04:35:38.360754 2677 raft_consensus.cc:2229] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:38.360971 2677 raft_consensus.cc:2258] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:38.361222 2678 tablet_replica.cc:330] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: stopping tablet replica
I20260620 04:35:38.361510 2678 raft_consensus.cc:2229] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:38.362658 2628 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Bootstrap starting.
I20260620 04:35:38.362844 2678 raft_consensus.cc:2258] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:38.363011 2677 ts_tablet_manager.cc:1905] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:38.363588 2628 tablet_bootstrap.cc:654] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.364554 2629 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Bootstrap starting.
I20260620 04:35:38.365401 2629 tablet_bootstrap.cc:654] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:38.365942 2677 ts_tablet_manager.cc:1918] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:38.366058 2677 log.cc:1196] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/0a5046857c064fb1b55d71b2fa188f92
I20260620 04:35:38.366869 2628 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: No bootstrap required, opened a new log
I20260620 04:35:38.366973 2628 ts_tablet_manager.cc:1397] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:35:38.367359 2678 ts_tablet_manager.cc:1905] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:38.367408 2628 raft_consensus.cc:348] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.367528 2628 raft_consensus.cc:374] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.367566 2628 raft_consensus.cc:729] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.367691 2628 consensus_queue.cc:260] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.367929 2628 ts_tablet_manager.cc:1428] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:38.368533 2677 ts_tablet_manager.cc:1939] T 0a5046857c064fb1b55d71b2fa188f92 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting consensus metadata
I20260620 04:35:38.369741 2107 catalog_manager.cc:4841] TS c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005): tablet 0a5046857c064fb1b55d71b2fa188f92 (table test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]) successfully deleted
I20260620 04:35:38.369956 2629 tablet_bootstrap.cc:492] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: No bootstrap required, opened a new log
I20260620 04:35:38.370044 2629 ts_tablet_manager.cc:1397] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260620 04:35:38.370461 2629 raft_consensus.cc:348] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.370575 2629 raft_consensus.cc:374] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:38.370611 2629 raft_consensus.cc:729] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Initialized, Role: FOLLOWER
I20260620 04:35:38.370723 2629 consensus_queue.cc:260] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.370965 2629 ts_tablet_manager.cc:1428] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 04:35:38.372871 2678 ts_tablet_manager.cc:1918] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:38.372977 2678 log.cc:1196] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/0a5046857c064fb1b55d71b2fa188f92
I20260620 04:35:38.373343 2678 ts_tablet_manager.cc:1939] T 0a5046857c064fb1b55d71b2fa188f92 P 06224bd9bd194073adc09fda4361ee51: Deleting consensus metadata
I20260620 04:35:38.374660 2109 catalog_manager.cc:4841] TS 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605): tablet 0a5046857c064fb1b55d71b2fa188f92 (table test-table [id=97b83b9ca6ed48ddb7eb91e5848ac25a]) successfully deleted
I20260620 04:35:38.510826 2636 raft_consensus.cc:482] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:38.510972 2636 raft_consensus.cc:504] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.512144 2636 leader_election.cc:288] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005)
I20260620 04:35:38.516506 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.512491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50385 (local address 127.0.98.3:39005)
0620 04:35:38.512611 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.512615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.512851 (+ 236us) server_negotiation.cc:407] Connection header received
0620 04:35:38.513009 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.513013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.513062 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.513144 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.513425 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.513989 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.515307 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.515885 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.516054 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.516141 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.516333 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.516405 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:35:38.516552 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.512413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.2:50385)
0620 04:35:38.512746 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.512758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.512899 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.513167 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.513170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.513187 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.513314 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.513319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.514100 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.514106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.515170 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.515190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.516030 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.516034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.516167 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.516216 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.516302 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.516379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":87,"threads_started":1}
I20260620 04:35:38.516921 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.512413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48997 (local address 127.0.98.1:46605)
0620 04:35:38.512555 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.512560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.513420 (+ 860us) server_negotiation.cc:407] Connection header received
0620 04:35:38.513655 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.513659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.513709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.513757 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:38.514115 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.514582 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.515667 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.516268 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.516361 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.516415 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.516735 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.516817 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:35:38.517078 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaac4b488064779bc6d5e4790c16c52" candidate_uuid: "92484a410dd744cebb7b25b76406a899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" is_pre_election: true
I20260620 04:35:38.517223 2559 raft_consensus.cc:2454] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92484a410dd744cebb7b25b76406a899 in term 0.
I20260620 04:35:38.517287 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.512431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46605 (local address 127.0.98.2:48997)
0620 04:35:38.512889 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.512904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.513495 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.513866 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.513868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.513881 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:38.513996 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.514001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.514691 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.514696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.515526 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.515535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.516480 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.516484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.516590 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.516610 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.517096 (+ 486us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.517180 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260620 04:35:38.517606 2351 leader_election.cc:302] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:38.517740 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaac4b488064779bc6d5e4790c16c52" candidate_uuid: "92484a410dd744cebb7b25b76406a899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51" is_pre_election: true
I20260620 04:35:38.517911 2636 raft_consensus.cc:2781] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:38.517915 2279 raft_consensus.cc:2454] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92484a410dd744cebb7b25b76406a899 in term 0.
I20260620 04:35:38.517968 2636 raft_consensus.cc:482] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:38.518011 2636 raft_consensus.cc:3037] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.519101 2636 raft_consensus.cc:504] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.519429 2636 leader_election.cc:288] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [CANDIDATE]: Term 1 election: Requested vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005)
I20260620 04:35:38.519752 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaac4b488064779bc6d5e4790c16c52" candidate_uuid: "92484a410dd744cebb7b25b76406a899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c689f3cd76ac4739b5c07cfaa686f77a"
I20260620 04:35:38.519796 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaac4b488064779bc6d5e4790c16c52" candidate_uuid: "92484a410dd744cebb7b25b76406a899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51"
I20260620 04:35:38.519886 2559 raft_consensus.cc:3037] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.519922 2279 raft_consensus.cc:3037] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:38.520835 2559 raft_consensus.cc:2454] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92484a410dd744cebb7b25b76406a899 in term 1.
I20260620 04:35:38.520920 2279 raft_consensus.cc:2454] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92484a410dd744cebb7b25b76406a899 in term 1.
I20260620 04:35:38.521239 2353 leader_election.cc:302] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06224bd9bd194073adc09fda4361ee51, 92484a410dd744cebb7b25b76406a899; no voters:
I20260620 04:35:38.521443 2636 raft_consensus.cc:2781] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:38.521652 2636 raft_consensus.cc:686] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 1 LEADER]: Becoming Leader. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Running, Role: LEADER
I20260620 04:35:38.521896 2636 consensus_queue.cc:237] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:38.524297 2122 catalog_manager.cc:5495] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92484a410dd744cebb7b25b76406a899 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "92484a410dd744cebb7b25b76406a899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: UNKNOWN } } }
W20260620 04:35:38.553800 2326 tablet.cc:2369] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:38.565171 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.560717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.0.98.2:34429)
0620 04:35:38.560878 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.560885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.560901 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:35:38.561097 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.561099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.561154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.561268 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.561563 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.562351 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.563409 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.563758 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.563852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.564134 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.564719 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.564828 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.564972 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.565032 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 04:35:38.578049 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.573811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36653 (local address 127.0.98.3:39005)
0620 04:35:38.573948 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.573952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.574136 (+ 184us) server_negotiation.cc:407] Connection header received
0620 04:35:38.574348 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.574351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.574398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.574482 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.574979 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.575603 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.576638 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.576941 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.577038 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.577438 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.577521 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.577594 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.577880 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.577937 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:35:38.578233 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.573644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.2:36653)
0620 04:35:38.574021 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.574045 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.574193 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.574653 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.574657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.574667 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:38.574779 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.574784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.575724 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.575727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.576460 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.576469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.577126 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.577133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.577222 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.577265 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.577706 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.577713 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.577811 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.577878 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
I20260620 04:35:38.584502 2279 raft_consensus.cc:1261] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Refusing update from remote peer 92484a410dd744cebb7b25b76406a899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.584522 2559 raft_consensus.cc:1261] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Refusing update from remote peer 92484a410dd744cebb7b25b76406a899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:38.584975 2683 consensus_queue.cc:1035] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.585220 2636 consensus_queue.cc:1035] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:38.603055 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.600073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46329 (local address 127.0.98.2:34429)
0620 04:35:38.600228 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:38.600235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:38.600248 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:35:38.600289 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:38.600296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:38.600333 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:38.600419 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:38.600652 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.601241 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.602077 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.602296 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.602387 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.602628 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.602688 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.602775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:38.602880 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 04:35:38.602926 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:35:38.603061 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:38.599710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34429 (local address 127.0.98.3:46329)
0620 04:35:38.599849 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:38.599864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:38.599939 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:38.600433 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:38.600436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:38.600444 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:38.600540 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.600546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.601344 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.601348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.601956 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:38.601965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:38.602399 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:38.602401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:38.602493 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:38.602524 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:38.602803 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:38.602809 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:35:38.602871 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:35:38.602950 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 04:35:38.603947 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:35:40.172914 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:40.173102 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:40.175812 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Sending DeleteTablet for 3 replicas of tablet acaac4b488064779bc6d5e4790c16c52
I20260620 04:35:40.176405 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet acaac4b488064779bc6d5e4790c16c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43266
I20260620 04:35:40.176553 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet acaac4b488064779bc6d5e4790c16c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37226
I20260620 04:35:40.176671 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet acaac4b488064779bc6d5e4790c16c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41042
I20260620 04:35:40.176822 2714 tablet_replica.cc:330] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: stopping tablet replica
I20260620 04:35:40.176954 2714 raft_consensus.cc:2229] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:40.177116 2714 raft_consensus.cc:2258] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:40.177368 2715 tablet_replica.cc:330] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: stopping tablet replica
I20260620 04:35:40.177367 2716 tablet_replica.cc:330] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: stopping tablet replica
I20260620 04:35:40.177508 2715 raft_consensus.cc:2229] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:40.177637 2716 raft_consensus.cc:2229] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:40.177809 2715 raft_consensus.cc:2258] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:40.177839 2716 raft_consensus.cc:2258] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:40.179207 2716 ts_tablet_manager.cc:1905] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:40.180575 2715 ts_tablet_manager.cc:1905] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:40.180671 2714 ts_tablet_manager.cc:1905] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 04:35:40 dist-test-slave-zmf4 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 04:35:40.196092 2715 ts_tablet_manager.cc:1918] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:35:40.196216 2715 log.cc:1196] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/acaac4b488064779bc6d5e4790c16c52
I20260620 04:35:40.196539 2715 ts_tablet_manager.cc:1939] T acaac4b488064779bc6d5e4790c16c52 P 92484a410dd744cebb7b25b76406a899: Deleting consensus metadata
I20260620 04:35:40.199011 2714 ts_tablet_manager.cc:1918] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:35:40.199115 2714 log.cc:1196] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/acaac4b488064779bc6d5e4790c16c52
I20260620 04:35:40.199409 2714 ts_tablet_manager.cc:1939] T acaac4b488064779bc6d5e4790c16c52 P 06224bd9bd194073adc09fda4361ee51: Deleting consensus metadata
Jun 20 04:35:40 dist-test-slave-zmf4 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:40.211797 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:40.198186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.0.98.62:39289)
0620 04:35:40.198443 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:40.198449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:40.198464 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:35:40.198529 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:40.198532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:40.198570 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:40.198645 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:40.199574 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.200127 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.201146 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.201401 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.201501 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:40.208100 (+ 6599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:40.208103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:40.208106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:40.208133 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:40.209969 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:40.210407 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:40.210410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:40.210412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:40.210497 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:40.210832 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:40.210835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:40.210836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:40.211036 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:40.211174 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:40.211443 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 04:35:40.211552 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260620 04:35:40.214584 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:40.215039 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:40.219813 2258 tablet_service.cc:1467] Processing CreateTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 (DEFAULT_TABLE table=test-table [id=f18947f79e634e2695fadddf82f13f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.219875 2399 tablet_service.cc:1467] Processing CreateTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 (DEFAULT_TABLE table=test-table [id=f18947f79e634e2695fadddf82f13f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.219966 2538 tablet_service.cc:1467] Processing CreateTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 (DEFAULT_TABLE table=test-table [id=f18947f79e634e2695fadddf82f13f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.220104 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce78cc6cd61647c1bd480fa22dc02b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.220104 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce78cc6cd61647c1bd480fa22dc02b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.222601 2725 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Bootstrap starting.
I20260620 04:35:40.222918 2726 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Bootstrap starting.
I20260620 04:35:40.223449 2725 tablet_bootstrap.cc:654] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.223863 2726 tablet_bootstrap.cc:654] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.224515 2725 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: No bootstrap required, opened a new log
I20260620 04:35:40.224588 2725 ts_tablet_manager.cc:1397] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:40.224941 2726 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: No bootstrap required, opened a new log
I20260620 04:35:40.225041 2726 ts_tablet_manager.cc:1397] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:40.225033 2725 raft_consensus.cc:348] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.225157 2725 raft_consensus.cc:374] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.225181 2725 raft_consensus.cc:729] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.225325 2725 consensus_queue.cc:260] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.225528 2726 raft_consensus.cc:348] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.225636 2726 raft_consensus.cc:374] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.225651 2725 ts_tablet_manager.cc:1428] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:40.225666 2726 raft_consensus.cc:729] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.225780 2726 consensus_queue.cc:260] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.226141 2726 ts_tablet_manager.cc:1428] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:40.229940 2107 catalog_manager.cc:4841] TS 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429): tablet acaac4b488064779bc6d5e4790c16c52 (table test-table [id=7df240935acc4374ae7eec8edd2801b9]) successfully deleted
I20260620 04:35:40.231807 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce78cc6cd61647c1bd480fa22dc02b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.231838 2716 ts_tablet_manager.cc:1918] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:35:40.231904 2716 log.cc:1196] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/acaac4b488064779bc6d5e4790c16c52
I20260620 04:35:40.232120 2109 catalog_manager.cc:4841] TS 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605): tablet acaac4b488064779bc6d5e4790c16c52 (table test-table [id=7df240935acc4374ae7eec8edd2801b9]) successfully deleted
I20260620 04:35:40.232277 2716 ts_tablet_manager.cc:1939] T acaac4b488064779bc6d5e4790c16c52 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting consensus metadata
I20260620 04:35:40.234426 2730 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Bootstrap starting.
I20260620 04:35:40.235265 2730 tablet_bootstrap.cc:654] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.236349 2730 tablet_bootstrap.cc:492] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: No bootstrap required, opened a new log
I20260620 04:35:40.236449 2730 ts_tablet_manager.cc:1397] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:40.236792 2730 raft_consensus.cc:348] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.236904 2730 raft_consensus.cc:374] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.236948 2730 raft_consensus.cc:729] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.237092 2730 consensus_queue.cc:260] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.237387 2730 ts_tablet_manager.cc:1428] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:35:40.266644 2107 catalog_manager.cc:4841] TS c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005): tablet acaac4b488064779bc6d5e4790c16c52 (table test-table [id=7df240935acc4374ae7eec8edd2801b9]) successfully deleted
I20260620 04:35:40.448982 2700 raft_consensus.cc:482] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:40.449141 2700 raft_consensus.cc:504] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.449666 2700 leader_election.cc:288] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:40.450327 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce78cc6cd61647c1bd480fa22dc02b43" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899" is_pre_election: true
I20260620 04:35:40.450348 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce78cc6cd61647c1bd480fa22dc02b43" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51" is_pre_election: true
I20260620 04:35:40.450501 2417 raft_consensus.cc:2454] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:40.450546 2279 raft_consensus.cc:2454] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:40.450865 2491 leader_election.cc:302] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:40.451040 2700 raft_consensus.cc:2781] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:40.451097 2700 raft_consensus.cc:482] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:40.451141 2700 raft_consensus.cc:3037] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.469518 2700 raft_consensus.cc:504] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.470206 2700 leader_election.cc:288] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Requested vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:40.470784 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce78cc6cd61647c1bd480fa22dc02b43" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899"
I20260620 04:35:40.470942 2417 raft_consensus.cc:3037] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.471599 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce78cc6cd61647c1bd480fa22dc02b43" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51"
I20260620 04:35:40.471729 2279 raft_consensus.cc:3037] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.472211 2417 raft_consensus.cc:2454] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:40.472558 2491 leader_election.cc:302] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:40.472728 2700 raft_consensus.cc:2781] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:40.472800 2700 raft_consensus.cc:686] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 LEADER]: Becoming Leader. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Running, Role: LEADER
I20260620 04:35:40.472958 2279 raft_consensus.cc:2454] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:40.472958 2700 consensus_queue.cc:237] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.475540 2122 catalog_manager.cc:5495] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a reported cstate change: term changed from 0 to 1, leader changed from <none> to c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: HEALTHY } } }
I20260620 04:35:40.523829 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:40.520349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.98.3:39005)
0620 04:35:40.520644 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:40.520659 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:40.520674 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:35:40.520712 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:40.520715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:40.520755 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:40.520847 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:40.521212 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.521749 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.522706 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.522967 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.523077 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:40.523320 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:40.523390 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:40.523450 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:40.523561 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 04:35:40.523606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260620 04:35:40.533187 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:40.529820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35857 (local address 127.0.98.3:39005)
0620 04:35:40.529936 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:40.529940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:40.530080 (+ 140us) server_negotiation.cc:407] Connection header received
0620 04:35:40.530144 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:40.530147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:40.530189 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:40.530262 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:40.530504 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.530974 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.531875 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.532113 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.532292 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:40.532549 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:40.532610 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:40.532687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:40.533039 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 04:35:40.533084 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:35:40.533212 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:40.529548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.3:35857)
0620 04:35:40.529981 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:40.529993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:40.530077 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:40.530298 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:40.530300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:40.530306 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:35:40.530393 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:40.530396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.531128 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.531130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:40.531728 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:40.531735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:40.532312 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:40.532319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:40.532390 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:40.532414 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:35:40.532748 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:35:40.532755 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:35:40.533010 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 04:35:40.533075 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":98,"threads_started":1}
I20260620 04:35:40.538674 2417 raft_consensus.cc:1261] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:40.538682 2279 raft_consensus.cc:1261] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:40.539039 2700 consensus_queue.cc:1035] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:40.539268 2634 consensus_queue.cc:1035] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:40.574208 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:40.574378 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:40.576495 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Sending DeleteTablet for 3 replicas of tablet ce78cc6cd61647c1bd480fa22dc02b43
I20260620 04:35:40.577039 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43266
I20260620 04:35:40.577267 2714 tablet_replica.cc:330] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: stopping tablet replica
I20260620 04:35:40.577315 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37226
I20260620 04:35:40.577414 2714 raft_consensus.cc:2229] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:40.577526 2714 raft_consensus.cc:2258] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:40.577740 2715 tablet_replica.cc:330] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: stopping tablet replica
I20260620 04:35:40.577872 2715 raft_consensus.cc:2229] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:40.578023 2715 raft_consensus.cc:2258] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:40.578469 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet ce78cc6cd61647c1bd480fa22dc02b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41042
I20260620 04:35:40.578631 2716 tablet_replica.cc:330] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: stopping tablet replica
I20260620 04:35:40.578605 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:35:40.578737 2716 raft_consensus.cc:2229] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:40.578779 2715 ts_tablet_manager.cc:1905] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:40.579003 2716 raft_consensus.cc:2258] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:35:40.579042 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:40.579859 2716 ts_tablet_manager.cc:1905] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:40.581007 2714 ts_tablet_manager.cc:1905] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:40.583732 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c9661e294615a3a560c35af628a5 (DEFAULT_TABLE table=test-table [id=e60ff9b5ec3e45b3a5e1bfe516ae629f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.583895 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c9661e294615a3a560c35af628a5 (DEFAULT_TABLE table=test-table [id=e60ff9b5ec3e45b3a5e1bfe516ae629f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.584120 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c9661e294615a3a560c35af628a5 (DEFAULT_TABLE table=test-table [id=e60ff9b5ec3e45b3a5e1bfe516ae629f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:40.649087 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c9661e294615a3a560c35af628a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.649084 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c9661e294615a3a560c35af628a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.652184 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c9661e294615a3a560c35af628a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:40.652915 2730 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Bootstrap starting.
I20260620 04:35:40.653955 2730 tablet_bootstrap.cc:654] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.654572 2716 ts_tablet_manager.cc:1918] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:40.654681 2716 log.cc:1196] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/ce78cc6cd61647c1bd480fa22dc02b43
I20260620 04:35:40.655098 2726 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: Bootstrap starting.
I20260620 04:35:40.655305 2716 ts_tablet_manager.cc:1939] T ce78cc6cd61647c1bd480fa22dc02b43 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting consensus metadata
I20260620 04:35:40.655619 2725 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Bootstrap starting.
I20260620 04:35:40.655963 2726 tablet_bootstrap.cc:654] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.656569 2725 tablet_bootstrap.cc:654] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:40.656807 2730 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: No bootstrap required, opened a new log
I20260620 04:35:40.656878 2730 ts_tablet_manager.cc:1397] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:35:40.657191 2726 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: No bootstrap required, opened a new log
I20260620 04:35:40.657269 2726 ts_tablet_manager.cc:1397] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:40.657253 2730 raft_consensus.cc:348] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.657379 2730 raft_consensus.cc:374] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.657408 2730 raft_consensus.cc:729] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.657514 2730 consensus_queue.cc:260] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.657775 2730 ts_tablet_manager.cc:1428] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:40.657706 2726 raft_consensus.cc:348] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.657827 2726 raft_consensus.cc:374] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.657869 2726 raft_consensus.cc:729] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06224bd9bd194073adc09fda4361ee51, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.658005 2726 consensus_queue.cc:260] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.658285 2726 ts_tablet_manager.cc:1428] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:40.658331 2725 tablet_bootstrap.cc:492] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: No bootstrap required, opened a new log
I20260620 04:35:40.658417 2725 ts_tablet_manager.cc:1397] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:40.658809 2725 raft_consensus.cc:348] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.658934 2725 raft_consensus.cc:374] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:40.658970 2725 raft_consensus.cc:729] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92484a410dd744cebb7b25b76406a899, State: Initialized, Role: FOLLOWER
I20260620 04:35:40.659101 2725 consensus_queue.cc:260] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.659351 2725 ts_tablet_manager.cc:1428] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:35:40.677210 2634 raft_consensus.cc:482] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:40.677376 2634 raft_consensus.cc:504] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.677851 2634 leader_election.cc:288] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:40.678442 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c9661e294615a3a560c35af628a5" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899" is_pre_election: true
I20260620 04:35:40.678408 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c9661e294615a3a560c35af628a5" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51" is_pre_election: true
I20260620 04:35:40.678609 2417 raft_consensus.cc:2454] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:40.678617 2279 raft_consensus.cc:2454] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 0.
I20260620 04:35:40.678911 2491 leader_election.cc:302] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92484a410dd744cebb7b25b76406a899, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:40.679177 2634 raft_consensus.cc:2781] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:40.679256 2634 raft_consensus.cc:482] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:40.679316 2634 raft_consensus.cc:3037] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.741639 2107 catalog_manager.cc:4841] TS c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005): tablet ce78cc6cd61647c1bd480fa22dc02b43 (table test-table [id=f18947f79e634e2695fadddf82f13f17]) successfully deleted
I20260620 04:35:40.746037 2634 raft_consensus.cc:504] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.746098 2715 ts_tablet_manager.cc:1918] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:40.746140 2714 ts_tablet_manager.cc:1918] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:35:40.746230 2714 log.cc:1196] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/ce78cc6cd61647c1bd480fa22dc02b43
I20260620 04:35:40.746228 2715 log.cc:1196] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/ce78cc6cd61647c1bd480fa22dc02b43
I20260620 04:35:40.746534 2714 ts_tablet_manager.cc:1939] T ce78cc6cd61647c1bd480fa22dc02b43 P 06224bd9bd194073adc09fda4361ee51: Deleting consensus metadata
I20260620 04:35:40.746539 2715 ts_tablet_manager.cc:1939] T ce78cc6cd61647c1bd480fa22dc02b43 P 92484a410dd744cebb7b25b76406a899: Deleting consensus metadata
I20260620 04:35:40.746941 2634 leader_election.cc:288] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Requested vote from peers 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605), 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429)
I20260620 04:35:40.746925 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c9661e294615a3a560c35af628a5" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06224bd9bd194073adc09fda4361ee51"
I20260620 04:35:40.747054 2279 raft_consensus.cc:3037] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.747524 2107 catalog_manager.cc:4841] TS 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429): tablet ce78cc6cd61647c1bd480fa22dc02b43 (table test-table [id=f18947f79e634e2695fadddf82f13f17]) successfully deleted
I20260620 04:35:40.747593 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c9661e294615a3a560c35af628a5" candidate_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92484a410dd744cebb7b25b76406a899"
I20260620 04:35:40.747713 2417 raft_consensus.cc:3037] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:40.748343 2279 raft_consensus.cc:2454] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:40.748679 2493 leader_election.cc:302] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06224bd9bd194073adc09fda4361ee51, c689f3cd76ac4739b5c07cfaa686f77a; no voters:
I20260620 04:35:40.748881 2634 raft_consensus.cc:2781] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:40.748879 2417 raft_consensus.cc:2454] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c689f3cd76ac4739b5c07cfaa686f77a in term 1.
I20260620 04:35:40.748963 2634 raft_consensus.cc:686] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 LEADER]: Becoming Leader. State: Replica: c689f3cd76ac4739b5c07cfaa686f77a, State: Running, Role: LEADER
I20260620 04:35:40.748968 2109 catalog_manager.cc:4841] TS 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605): tablet ce78cc6cd61647c1bd480fa22dc02b43 (table test-table [id=f18947f79e634e2695fadddf82f13f17]) successfully deleted
I20260620 04:35:40.749106 2634 consensus_queue.cc:237] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260620 04:35:40.750872 2122 catalog_manager.cc:5495] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a reported cstate change: term changed from 0 to 1, leader changed from <none> to c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c689f3cd76ac4739b5c07cfaa686f77a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: HEALTHY } } }
I20260620 04:35:40.787801 2417 raft_consensus.cc:1261] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:40.788213 2279 raft_consensus.cc:1261] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51 [term 1 FOLLOWER]: Refusing update from remote peer c689f3cd76ac4739b5c07cfaa686f77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:40.788158 2634 consensus_queue.cc:1035] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92484a410dd744cebb7b25b76406a899" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:40.788522 2634 consensus_queue.cc:1035] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06224bd9bd194073adc09fda4361ee51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:35:40.801136 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:35:42.354152 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:42.354292 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:42.356503 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43a024b77ab4ca68f40272498b3865f: Sending DeleteTablet for 3 replicas of tablet 9e07c9661e294615a3a560c35af628a5
I20260620 04:35:42.357012 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c9661e294615a3a560c35af628a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43266
I20260620 04:35:42.357153 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c9661e294615a3a560c35af628a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37226
I20260620 04:35:42.357498 2768 tablet_replica.cc:330] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: stopping tablet replica
I20260620 04:35:42.357491 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c9661e294615a3a560c35af628a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41042
I20260620 04:35:42.357651 2768 raft_consensus.cc:2229] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:42.357759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2194
I20260620 04:35:42.357780 2768 raft_consensus.cc:2258] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:42.357843 2767 tablet_replica.cc:330] T 9e07c9661e294615a3a560c35af628a5 P 06224bd9bd194073adc09fda4361ee51: stopping tablet replica
I20260620 04:35:42.358788 2769 tablet_replica.cc:330] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: stopping tablet replica
I20260620 04:35:42.358961 2768 ts_tablet_manager.cc:1905] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:42.358999 2769 raft_consensus.cc:2229] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:42.359228 2769 raft_consensus.cc:2258] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:42.359913 2769 ts_tablet_manager.cc:1905] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:42.362571 2768 ts_tablet_manager.cc:1918] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:35:42.362674 2768 log.cc:1196] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/9e07c9661e294615a3a560c35af628a5
I20260620 04:35:42.362635 2769 ts_tablet_manager.cc:1918] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:35:42.362773 2769 log.cc:1196] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/9e07c9661e294615a3a560c35af628a5
I20260620 04:35:42.362974 2768 ts_tablet_manager.cc:1939] T 9e07c9661e294615a3a560c35af628a5 P 92484a410dd744cebb7b25b76406a899: Deleting consensus metadata
I20260620 04:35:42.363118 2769 ts_tablet_manager.cc:1939] T 9e07c9661e294615a3a560c35af628a5 P c689f3cd76ac4739b5c07cfaa686f77a: Deleting consensus metadata
I20260620 04:35:42.363883 2107 catalog_manager.cc:4841] TS c689f3cd76ac4739b5c07cfaa686f77a (127.0.98.3:39005): tablet 9e07c9661e294615a3a560c35af628a5 (table test-table [id=e60ff9b5ec3e45b3a5e1bfe516ae629f]) successfully deleted
I20260620 04:35:42.364444 2107 catalog_manager.cc:4841] TS 92484a410dd744cebb7b25b76406a899 (127.0.98.2:34429): tablet 9e07c9661e294615a3a560c35af628a5 (table test-table [id=e60ff9b5ec3e45b3a5e1bfe516ae629f]) successfully deleted
W20260620 04:35:42.368094 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46605 (error 108)
I20260620 04:35:42.368306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2334
I20260620 04:35:42.369797 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:42.369369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46605 (local address 127.0.0.1:43292)
0620 04:35:42.369650 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:42.369717 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
W20260620 04:35:42.370046 2109 catalog_manager.cc:4584] TS 06224bd9bd194073adc09fda4361ee51 (127.0.98.1:46605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e07c9661e294615a3a560c35af628a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46605: connect: Connection refused (error 111)
I20260620 04:35:42.376403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2474
I20260620 04:35:42.383071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2090
2026-06-20T04:35:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6484 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:42 dist-test-slave-zmf4 krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:42 dist-test-slave-zmf4 krb5kdc[2774](info): set up 2 sockets
Jun 20 04:35:42 dist-test-slave-zmf4 krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260620 04:35:43.460093 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:43 dist-test-slave-zmf4 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:43.584518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:43.698385 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:43.698760 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:43.698829 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:43.703033 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:43.703095 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:43.703115 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:43.703131 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:43.703172 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:43.708496 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:43.709517 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:43.710522 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:43.719235 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:43.719266 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:43.719290 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:43.719480 2787 server_base.cc:1034] running on GCE node
I20260620 04:35:43.720139 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:43.721237 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:43.722445 2787 hybrid_clock.cc:648] HybridClock initialized: now 1781930143722379 us; error 77 us; skew 500 ppm
Jun 20 04:35:43 dist-test-slave-zmf4 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:43.725780 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:43.726698 2787 webserver.cc:460] Webserver started at http://127.0.98.62:32949/ using document root <none> and password file <none>
I20260620 04:35:43.727294 2787 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:43.727391 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:43.727648 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:43.729882 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "f7daba70df244d1687c7b465e0bde74c"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "9470da0e597ab7a55fb0661fb63f9dbb"
server_key_iv: "2b5ae9d62d57a96e7ed082b53a03e87c"
server_key_version: "encryptionkey@0"
I20260620 04:35:43.730504 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7daba70df244d1687c7b465e0bde74c"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "9470da0e597ab7a55fb0661fb63f9dbb"
server_key_iv: "2b5ae9d62d57a96e7ed082b53a03e87c"
server_key_version: "encryptionkey@0"
I20260620 04:35:43.734401 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 04:35:43.736902 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:43.738157 2787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:43.738306 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "f7daba70df244d1687c7b465e0bde74c"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "9470da0e597ab7a55fb0661fb63f9dbb"
server_key_iv: "2b5ae9d62d57a96e7ed082b53a03e87c"
server_key_version: "encryptionkey@0"
I20260620 04:35:43.738417 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:43.756451 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:43.759605 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:43.759807 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:43.766523 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42447
I20260620 04:35:43.766587 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42447 every 8 connection(s)
I20260620 04:35:43.767447 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:43.770251 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:43.770956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2787
I20260620 04:35:43.771044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:43.771312 392 external_mini_cluster.cc:1428] Setting key be5af02473509d8f759a4c359c15b791
I20260620 04:35:43.776594 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c: Bootstrap starting.
I20260620 04:35:43.779076 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:43.779878 2855 log.cc:824] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:35:43 dist-test-slave-zmf4 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:43.782680 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c: No bootstrap required, opened a new log
I20260620 04:35:43.786461 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:43.772595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.62:42447)
0620 04:35:43.773002 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:43.773021 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:43.773096 (+ 75us) server_negotiation.cc:407] Connection header received
0620 04:35:43.773970 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:43.773993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:43.774380 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:43.774746 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:43.775434 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:43.776867 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:43.777719 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:43.778357 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:43.778513 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:43.781201 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:43.781207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:43.781223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:43.781256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:43.783323 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:43.783753 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:43.783759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:43.783765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:43.783891 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:43.784216 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:43.784219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:43.784221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:43.784730 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:43.784959 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:43.785251 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 04:35:43.785537 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260620 04:35:43.787206 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260620 04:35:43.787463 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:43.787515 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7daba70df244d1687c7b465e0bde74c, State: Initialized, Role: FOLLOWER
I20260620 04:35:43.787969 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260620 04:35:43.788101 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:43.788183 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:43.788347 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:43.789393 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260620 04:35:43.789728 2855 leader_election.cc:302] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7daba70df244d1687c7b465e0bde74c; no voters:
I20260620 04:35:43.790014 2855 leader_election.cc:288] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:43.790176 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:43.790416 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [term 1 LEADER]: Becoming Leader. State: Replica: f7daba70df244d1687c7b465e0bde74c, State: Running, Role: LEADER
I20260620 04:35:43.790712 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } }
I20260620 04:35:43.791170 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:43.792498 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7daba70df244d1687c7b465e0bde74c. Latest consensus state: current_term: 1 leader_uuid: "f7daba70df244d1687c7b465e0bde74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } } }
I20260620 04:35:43.792510 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7daba70df244d1687c7b465e0bde74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7daba70df244d1687c7b465e0bde74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42447 } } }
I20260620 04:35:43.792644 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:43.792654 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:43.793018 2870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:43.795542 2870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:43.800827 2870 catalog_manager.cc:1347] Generated new cluster ID: 7870e9e4816e46d9a626e7c422bfdcd1
I20260620 04:35:43.800904 2870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:43.811692 2870 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:43.812824 2870 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:43.818264 2870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7daba70df244d1687c7b465e0bde74c: Generated new TSK 0
I20260620 04:35:43.818935 2870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:43.848491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:43.968034 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:43.968350 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:43.968395 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:43.972350 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:43.972405 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:43.972460 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:43.978193 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:43.979305 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:43.980383 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:43.990604 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:43.990813 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:43.990828 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:43.991134 2881 server_base.cc:1034] running on GCE node
I20260620 04:35:43.991848 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:43.992645 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:43.993884 2881 hybrid_clock.cc:648] HybridClock initialized: now 1781930143993864 us; error 30 us; skew 500 ppm
Jun 20 04:35:43 dist-test-slave-zmf4 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:43.996801 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:43.997669 2881 webserver.cc:460] Webserver started at http://127.0.98.1:40061/ using document root <none> and password file <none>
I20260620 04:35:43.998226 2881 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:43.998317 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:43.998548 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:44.000720 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f9e115d52584081bc92626e26f222d5"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "ec1d187a0f219eb0bd97a05bcaec5cb6"
server_key_iv: "983c7214715a81c4a92965858ddf3f7f"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.001257 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f9e115d52584081bc92626e26f222d5"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "ec1d187a0f219eb0bd97a05bcaec5cb6"
server_key_iv: "983c7214715a81c4a92965858ddf3f7f"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.005115 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:35:44.007761 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.009200 2881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:44.009342 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3f9e115d52584081bc92626e26f222d5"
format_stamp: "Formatted at 2026-06-20 04:35:43 on dist-test-slave-zmf4"
server_key: "ec1d187a0f219eb0bd97a05bcaec5cb6"
server_key_iv: "983c7214715a81c4a92965858ddf3f7f"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.009455 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:44.024971 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:44.028051 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:44.028323 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:44.028885 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:44.029829 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:44.029924 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.030004 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:44.030062 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.040341 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42361
I20260620 04:35:44.040385 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42361 every 8 connection(s)
I20260620 04:35:44.041224 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:44.045429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 2881
I20260620 04:35:44.045567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:44.045868 392 external_mini_cluster.cc:1428] Setting key c6373250250bb49a97bd8a71e0c6769c
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:44.055662 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.043139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40835 (local address 127.0.98.62:42447)
0620 04:35:44.043358 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:44.043363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:44.044132 (+ 769us) server_negotiation.cc:407] Connection header received
0620 04:35:44.045516 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:44.045521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:44.045580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:44.045682 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:44.046818 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.047354 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.048533 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.048767 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.048880 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.051702 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:44.051711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:44.051713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:44.051737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:44.053550 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.054089 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.054093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.054095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.054207 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.054548 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.054552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.054554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.054735 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:44.054838 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:44.055370 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 04:35:44.055491 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:35:44.056589 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.043527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.1:40835)
0620 04:35:44.043974 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:44.044022 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:44.045313 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:44.045822 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:44.045828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:44.046212 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:44.046671 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.046685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.047482 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.047485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.048415 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.048424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.048913 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.048917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.049078 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.049667 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:44.049683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:44.051501 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:44.053701 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.053707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.053722 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.053981 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.054313 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.054316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.054318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.054454 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.054847 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:44.054856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:44.055147 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 04:35:44.055332 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 04:35:44.055647 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260620 04:35:44.057878 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260620 04:35:44.058115 3010 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:44.058633 3010 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
I20260620 04:35:44.060381 2819 ts_manager.cc:194] Registered new tserver with Master: 3f9e115d52584081bc92626e26f222d5 (127.0.98.1:42361)
I20260620 04:35:44.062260 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:44.075690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:44.186201 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:44.186527 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:44.186586 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:44.190665 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:44.190721 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:44.190776 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:44.196430 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:44.197458 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:44.198478 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:44.208467 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:44.208438 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:44.208602 3015 server_base.cc:1034] running on GCE node
W20260620 04:35:44.208455 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:44.209115 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:44.209870 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:44.211077 3015 hybrid_clock.cc:648] HybridClock initialized: now 1781930144211070 us; error 40 us; skew 500 ppm
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:44.214262 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:44.215163 3015 webserver.cc:460] Webserver started at http://127.0.98.2:36121/ using document root <none> and password file <none>
I20260620 04:35:44.215773 3015 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:44.215871 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:44.216117 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:44.218308 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "8506e60b5d424f509183c7e3703af286"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "4f7766b3e15b9f93fb728bc48d1aee6d"
server_key_iv: "513ce52891be0e864afde56b3503ec2a"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.218915 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8506e60b5d424f509183c7e3703af286"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "4f7766b3e15b9f93fb728bc48d1aee6d"
server_key_iv: "513ce52891be0e864afde56b3503ec2a"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.222918 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:35:44.225559 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.226754 3015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:44.226903 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "8506e60b5d424f509183c7e3703af286"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "4f7766b3e15b9f93fb728bc48d1aee6d"
server_key_iv: "513ce52891be0e864afde56b3503ec2a"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.227015 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:44.238498 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:44.241827 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:44.242017 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:44.242697 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:44.243474 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:44.243521 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.243568 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:44.243626 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.252569 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37771
I20260620 04:35:44.252607 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37771 every 8 connection(s)
I20260620 04:35:44.253439 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:44.261802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3015
I20260620 04:35:44.261952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:44.262231 392 external_mini_cluster.cc:1428] Setting key 655d4c99cb71b5b9d158a1eea730c447
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:44.268515 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.255795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49933 (local address 127.0.98.62:42447)
0620 04:35:44.255951 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:44.255956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:44.256761 (+ 805us) server_negotiation.cc:407] Connection header received
0620 04:35:44.258263 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:44.258268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:44.258324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:44.258411 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:44.259593 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.260187 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.261448 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.261710 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.261831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.264684 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:44.264689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:44.264692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:44.264717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:44.266219 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.266732 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.266737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.266739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.266820 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.267144 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.267149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.267151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.267326 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:44.267448 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:44.268128 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 04:35:44.268293 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:35:44.269240 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.256081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.2:49933)
0620 04:35:44.256595 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:44.256641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:44.258039 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:44.258581 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:44.258588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:44.258965 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:44.259440 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.259454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.260364 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.260375 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.261311 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.261320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.261868 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.261873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.262077 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.262773 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:44.262796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:44.264515 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:44.266348 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.266354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.266368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.266627 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.266916 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.266918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.266920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.267049 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.267473 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:44.267483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:44.267811 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 04:35:44.268038 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 04:35:44.268389 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":116,"threads_started":1}
I20260620 04:35:44.270351 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260620 04:35:44.270596 3144 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:44.271176 3144 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:35:44.272360 2819 ts_manager.cc:194] Registered new tserver with Master: 8506e60b5d424f509183c7e3703af286 (127.0.98.2:37771)
I20260620 04:35:44.273032 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:44.290988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:44.401890 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:44.402143 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:44.402187 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:44.406234 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:44.406291 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:44.406348 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:44.411459 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:44.412369 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:44.413327 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:44.422546 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:44.422570 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:44.422546 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:44.422816 3149 server_base.cc:1034] running on GCE node
I20260620 04:35:44.423244 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:44.423894 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:44.425113 3149 hybrid_clock.cc:648] HybridClock initialized: now 1781930144425086 us; error 53 us; skew 500 ppm
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:44.428117 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:44.429018 3149 webserver.cc:460] Webserver started at http://127.0.98.3:36671/ using document root <none> and password file <none>
I20260620 04:35:44.429574 3149 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:44.429699 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:44.429940 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:44.431986 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d2853f2197840a2bb6738d33dd3d7ac"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "f3b2273f736f6210b172eba0fcdfd8a9"
server_key_iv: "8229fb7f5eaac06a8f5705fffb3fa969"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.432610 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d2853f2197840a2bb6738d33dd3d7ac"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "f3b2273f736f6210b172eba0fcdfd8a9"
server_key_iv: "8229fb7f5eaac06a8f5705fffb3fa969"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.436137 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:44.438619 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.439738 3149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:35:44.439826 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "3d2853f2197840a2bb6738d33dd3d7ac"
format_stamp: "Formatted at 2026-06-20 04:35:44 on dist-test-slave-zmf4"
server_key: "f3b2273f736f6210b172eba0fcdfd8a9"
server_key_iv: "8229fb7f5eaac06a8f5705fffb3fa969"
server_key_version: "encryptionkey@0"
I20260620 04:35:44.439949 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:44.466470 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:44.471148 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:44.471374 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:44.472190 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:44.473316 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:44.473384 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.473443 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:44.473477 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:44.482548 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36423
I20260620 04:35:44.482574 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36423 every 8 connection(s)
I20260620 04:35:44.483376 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:44.487498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3149
I20260620 04:35:44.487615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:44.487959 392 external_mini_cluster.cc:1428] Setting key d9980d155945483a9b58c18ad6f5f283
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:44.497956 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.485259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45083 (local address 127.0.98.62:42447)
0620 04:35:44.485413 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:44.485417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:44.486319 (+ 902us) server_negotiation.cc:407] Connection header received
0620 04:35:44.487947 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:44.487950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:44.487998 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:44.488132 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:44.489456 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.490037 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.491371 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.491615 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.491733 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.494285 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:44.494289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:44.494291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:44.494312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:44.495769 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.496428 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.496431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.496432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.496506 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:44.496903 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:44.496906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:44.496907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:44.497142 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:44.497225 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:44.497728 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 04:35:44.497828 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:35:44.498699 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:44.485593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42447 (local address 127.0.98.3:45083)
0620 04:35:44.486163 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:44.486206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:44.487682 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:44.488442 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:44.488449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:44.488823 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:44.489303 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.489316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.490172 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.490178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.491238 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:44.491248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:44.491768 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:44.491772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:44.491957 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:44.492593 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:44.492617 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:44.494106 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:44.495900 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.495905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.495925 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.496232 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.496633 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:44.496636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:44.496637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:44.496770 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:44.497254 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:44.497259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:44.497481 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 04:35:44.497635 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 04:35:44.497866 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":198,"threads_started":1}
I20260620 04:35:44.499914 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42447
I20260620 04:35:44.500216 3278 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:44.500763 3278 heartbeater.cc:511] Master 127.0.98.62:42447 requested a full tablet report, sending...
I20260620 04:35:44.501732 2818 ts_manager.cc:194] Registered new tserver with Master: 3d2853f2197840a2bb6738d33dd3d7ac (127.0.98.3:36423)
I20260620 04:35:44.502321 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45083
I20260620 04:35:44.513880 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:35:44.526955 2858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:44.525716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.0.98.62:42447)
0620 04:35:44.525838 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:44.525842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:44.525923 (+ 81us) server_negotiation.cc:407] Connection header received
0620 04:35:44.526015 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:44.526018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:44.526056 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:44.526129 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:44.526851 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38222: BlockingRecv error: recv got EOF from 127.0.0.1:38222 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:44.528438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2881
I20260620 04:35:44.533900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3015
I20260620 04:35:44.539196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3149
I20260620 04:35:44.544504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 2787
2026-06-20T04:35:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2158 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[3290](info): set up 2 sockets
Jun 20 04:35:44 dist-test-slave-zmf4 krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260620 04:35:45.624153 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:45 dist-test-slave-zmf4 krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:45.746781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:45.860929 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:45.861191 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:45.861230 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:45.865326 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:45.865387 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:45.865408 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:45.865424 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:45.865439 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:45.870679 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:45.871683 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:45.872718 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:45.881335 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:45.881393 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:45.881791 3303 server_base.cc:1034] running on GCE node
W20260620 04:35:45.881335 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:45.882592 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:45.883934 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:45.885139 3303 hybrid_clock.cc:648] HybridClock initialized: now 1781930145885108 us; error 44 us; skew 500 ppm
Jun 20 04:35:45 dist-test-slave-zmf4 krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:45.892527 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:45.893442 3303 webserver.cc:460] Webserver started at http://127.0.98.62:34305/ using document root <none> and password file <none>
I20260620 04:35:45.894125 3303 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:45.894203 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:45.894418 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:45.896919 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "fc4d116cbaf448ec93b9f3549726b8c0"
format_stamp: "Formatted at 2026-06-20 04:35:45 on dist-test-slave-zmf4"
server_key: "d1b91cc0ec081a10de23b67bb285f517"
server_key_iv: "adc1955be6b383604911361f0ff73f8c"
server_key_version: "encryptionkey@0"
I20260620 04:35:45.897697 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc4d116cbaf448ec93b9f3549726b8c0"
format_stamp: "Formatted at 2026-06-20 04:35:45 on dist-test-slave-zmf4"
server_key: "d1b91cc0ec081a10de23b67bb285f517"
server_key_iv: "adc1955be6b383604911361f0ff73f8c"
server_key_version: "encryptionkey@0"
I20260620 04:35:45.901568 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:35:45.904107 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:45.905589 3303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:35:45.905779 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "fc4d116cbaf448ec93b9f3549726b8c0"
format_stamp: "Formatted at 2026-06-20 04:35:45 on dist-test-slave-zmf4"
server_key: "d1b91cc0ec081a10de23b67bb285f517"
server_key_iv: "adc1955be6b383604911361f0ff73f8c"
server_key_version: "encryptionkey@0"
I20260620 04:35:45.905896 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:45.920360 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:45.923998 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:45.924312 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:45.931493 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38551
I20260620 04:35:45.931811 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38551 every 8 connection(s)
I20260620 04:35:45.932510 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:45.933187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3303
I20260620 04:35:45.933339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:45.933761 392 external_mini_cluster.cc:1428] Setting key fb9336eac622303af4099c5198afdf3d
I20260620 04:35:45.937124 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 04:35:45 dist-test-slave-zmf4 krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:45.942721 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: Bootstrap starting.
I20260620 04:35:45.944970 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:45.945688 3371 log.cc:824] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:45.947649 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: No bootstrap required, opened a new log
I20260620 04:35:45.948405 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:45.935212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.0.98.62:38551)
0620 04:35:45.935652 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:45.935672 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:45.935707 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:35:45.936526 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:45.936545 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:45.936884 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:45.937232 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:45.937993 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:45.939010 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:45.939883 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:45.940422 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:45.940587 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:45.943318 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:45.943324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:45.943336 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:45.943366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:45.945343 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:45.945825 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:45.945839 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:45.945843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:45.945956 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:45.946273 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:45.946277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:45.946279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:45.946688 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:45.946913 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:45.947234 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 04:35:45.947529 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260620 04:35:45.952428 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } }
I20260620 04:35:45.952759 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:45.952833 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4d116cbaf448ec93b9f3549726b8c0, State: Initialized, Role: FOLLOWER
I20260620 04:35:45.953359 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } }
I20260620 04:35:45.953550 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:45.953629 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:45.953750 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:45.953991 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:35:45.954886 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } }
I20260620 04:35:45.955276 3371 leader_election.cc:302] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4d116cbaf448ec93b9f3549726b8c0; no voters:
I20260620 04:35:45.955631 3371 leader_election.cc:288] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:45.955850 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:45.956135 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [term 1 LEADER]: Becoming Leader. State: Replica: fc4d116cbaf448ec93b9f3549726b8c0, State: Running, Role: LEADER
I20260620 04:35:45.956475 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } }
I20260620 04:35:45.957060 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:45.958226 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } } }
I20260620 04:35:45.958349 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:45.958581 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4d116cbaf448ec93b9f3549726b8c0. Latest consensus state: current_term: 1 leader_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4d116cbaf448ec93b9f3549726b8c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38551 } } }
I20260620 04:35:45.958668 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0 [sys.catalog]: This master's current role is: LEADER
W20260620 04:35:45.962105 3392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:35:45.962201 3392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:35:45.962785 3386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:45.963613 3386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:45.969581 3386 catalog_manager.cc:1347] Generated new cluster ID: 0c40f9a07c254fbfa8db38d213cc10e4
I20260620 04:35:45.969656 3386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:45.975692 3386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:45.976965 3386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:45.982221 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:45.976484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.98.62:38551)
0620 04:35:45.976587 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:45.976592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:45.976863 (+ 271us) server_negotiation.cc:407] Connection header received
0620 04:35:45.976985 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:45.976988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:45.977049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:45.977141 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:45.977855 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:45.978669 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:45.979823 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:45.980240 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:45.980339 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:45.981236 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:45.981240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:45.981243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:45.981787 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:45.981819 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:45.981837 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:45.981925 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:45.982027 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 04:35:45.982086 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:35:45.985523 3386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4d116cbaf448ec93b9f3549726b8c0: Generated new TSK 0
I20260620 04:35:45.986169 3386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:35:45.991545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3303
2026-06-20T04:35:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1445 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:46 dist-test-slave-zmf4 krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:46 dist-test-slave-zmf4 krb5kdc[3404](info): set up 2 sockets
Jun 20 04:35:46 dist-test-slave-zmf4 krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260620 04:35:47.068377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:47.196652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:47.311437 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:47.311877 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:47.311939 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:47.315853 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:47.315901 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:47.315922 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:47.315937 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:47.315953 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:47.321178 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:47.322129 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:47.323084 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:35:47.331857 3417 server_base.cc:1034] running on GCE node
W20260620 04:35:47.331931 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:47.331892 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:47.331991 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:47.332653 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:47.333825 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:47.335050 3417 hybrid_clock.cc:648] HybridClock initialized: now 1781930147335012 us; error 40 us; skew 500 ppm
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:47.337924 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:47.338807 3417 webserver.cc:460] Webserver started at http://127.0.98.62:34439/ using document root <none> and password file <none>
I20260620 04:35:47.339396 3417 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:47.339495 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:47.339733 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:47.341917 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "e1911c43ddae491a8db43988b57f73ff"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "53804aeaf9fa9121e565e5f91fbe55e4"
server_key_iv: "1b94881bf7db92ed567f833ecbf23b7c"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.342581 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1911c43ddae491a8db43988b57f73ff"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "53804aeaf9fa9121e565e5f91fbe55e4"
server_key_iv: "1b94881bf7db92ed567f833ecbf23b7c"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.346499 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 04:35:47.348929 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.350116 3417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:47.350256 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "e1911c43ddae491a8db43988b57f73ff"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "53804aeaf9fa9121e565e5f91fbe55e4"
server_key_iv: "1b94881bf7db92ed567f833ecbf23b7c"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.350368 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:47.362458 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:47.365665 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:47.365869 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:47.373271 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46283
I20260620 04:35:47.373620 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46283 every 8 connection(s)
I20260620 04:35:47.374243 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:47.376925 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:47.381986 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff: Bootstrap starting.
I20260620 04:35:47.382889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3417
I20260620 04:35:47.383039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:47.383327 392 external_mini_cluster.cc:1428] Setting key 79aa60c0d3d0bb0bcf4fcfd335947fce
I20260620 04:35:47.384605 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:47.385317 3485 log.cc:824] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:47.387419 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff: No bootstrap required, opened a new log
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:47.392028 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } }
I20260620 04:35:47.392305 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:47.392366 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1911c43ddae491a8db43988b57f73ff, State: Initialized, Role: FOLLOWER
I20260620 04:35:47.392786 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } }
I20260620 04:35:47.392931 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:47.393019 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:47.393129 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:47.394227 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } }
I20260620 04:35:47.394567 3485 leader_election.cc:302] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1911c43ddae491a8db43988b57f73ff; no voters:
I20260620 04:35:47.394912 3485 leader_election.cc:288] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:47.394999 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:47.395195 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [term 1 LEADER]: Becoming Leader. State: Replica: e1911c43ddae491a8db43988b57f73ff, State: Running, Role: LEADER
I20260620 04:35:47.395524 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } }
I20260620 04:35:47.396147 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:47.397136 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1911c43ddae491a8db43988b57f73ff. Latest consensus state: current_term: 1 leader_uuid: "e1911c43ddae491a8db43988b57f73ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } } }
I20260620 04:35:47.397264 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:47.397606 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1911c43ddae491a8db43988b57f73ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1911c43ddae491a8db43988b57f73ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46283 } } }
I20260620 04:35:47.397698 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:47.398005 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:47.398928 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:47.384409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.98.62:46283)
0620 04:35:47.384805 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:47.384820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:47.384859 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:35:47.385498 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:47.385522 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:47.385835 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:47.386096 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:47.387049 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.388084 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.389287 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.389761 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.389888 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:47.392865 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:47.392870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:47.392882 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:47.392906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:47.395106 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.395641 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.395649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.395656 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.395793 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.396085 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.396088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.396089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.396589 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:47.397442 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:47.397581 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 04:35:47.397867 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260620 04:35:47.400950 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:47.406276 3499 catalog_manager.cc:1347] Generated new cluster ID: 303384b3635c48bcb5bc4778149ba9b7
I20260620 04:35:47.406342 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:47.425369 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:47.426301 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:47.434332 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1911c43ddae491a8db43988b57f73ff: Generated new TSK 0
I20260620 04:35:47.435011 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:47.463923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:47.576148 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:47.576508 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:47.576579 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:47.580941 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:47.580994 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:47.581049 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:47.585987 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:47.586900 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:47.587829 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:47.597321 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:47.597410 3511 server_base.cc:1034] running on GCE node
W20260620 04:35:47.597327 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:47.597513 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:47.597864 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:47.598500 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:47.599689 3511 hybrid_clock.cc:648] HybridClock initialized: now 1781930147599673 us; error 28 us; skew 500 ppm
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:47.602499 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:47.603291 3511 webserver.cc:460] Webserver started at http://127.0.98.1:43643/ using document root <none> and password file <none>
I20260620 04:35:47.603827 3511 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:47.603914 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:47.604135 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:47.606082 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "a18d9357c9b3443a9639a6d5de3bbd0e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "fbf34a3ac346c46f958a4180bb55edc6"
server_key_iv: "d932f910722761d0c67968d64f262fd9"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.606622 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a18d9357c9b3443a9639a6d5de3bbd0e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "fbf34a3ac346c46f958a4180bb55edc6"
server_key_iv: "d932f910722761d0c67968d64f262fd9"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.610288 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:35:47.612771 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.613946 3511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:47.614089 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "a18d9357c9b3443a9639a6d5de3bbd0e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "fbf34a3ac346c46f958a4180bb55edc6"
server_key_iv: "d932f910722761d0c67968d64f262fd9"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.614197 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:47.625135 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:47.628398 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:47.628607 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:47.629216 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:47.630055 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:47.630144 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.630223 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:47.630263 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.639567 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39257
I20260620 04:35:47.639647 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39257 every 8 connection(s)
I20260620 04:35:47.640590 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:47.649935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3511
I20260620 04:35:47.650061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:47.650379 392 external_mini_cluster.cc:1428] Setting key d1d96010e96cee45bfa06baa917fc7ec
I20260620 04:35:47.654412 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:47.642388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55015 (local address 127.0.98.62:46283)
0620 04:35:47.642528 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:47.642532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:47.643246 (+ 714us) server_negotiation.cc:407] Connection header received
0620 04:35:47.644683 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:47.644686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:47.644745 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:47.644846 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:47.646109 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.646633 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.647862 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.648118 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.648301 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:47.650900 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:47.650904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:47.650907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:47.650938 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:47.652461 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.652936 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.652941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.652943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.653023 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.653365 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.653370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.653371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.653543 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:47.653652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:47.654108 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 04:35:47.654226 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:35:47.655203 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:47.642634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46283 (local address 127.0.98.1:55015)
0620 04:35:47.643088 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:47.643135 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:47.644482 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:47.645007 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:47.645014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:47.645357 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:47.645930 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:47.645946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.646770 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.646773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:47.647723 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:47.647732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.648329 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.648332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:47.648478 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:47.649046 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:47.649063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:47.650737 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:47.652571 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:47.652580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:47.652593 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:47.652835 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:47.653114 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:47.653118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:47.653120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:47.653249 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:47.653650 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:47.653656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:47.653890 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:35:47.654076 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 04:35:47.654370 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20260620 04:35:47.656589 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46283
I20260620 04:35:47.656836 3640 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:47.657300 3640 heartbeater.cc:511] Master 127.0.98.62:46283 requested a full tablet report, sending...
I20260620 04:35:47.658771 3449 ts_manager.cc:194] Registered new tserver with Master: a18d9357c9b3443a9639a6d5de3bbd0e (127.0.98.1:39257)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:47.659886 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:47.675081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:47.781946 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:47.782177 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:47.782218 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:47.786230 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:47.786283 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:47.786365 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:47.791472 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:47.792496 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:47.793457 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:47.802815 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:47.802811 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:47.802806 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:47.803180 3645 server_base.cc:1034] running on GCE node
I20260620 04:35:47.803558 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:47.804245 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:47.805480 3645 hybrid_clock.cc:648] HybridClock initialized: now 1781930147805413 us; error 73 us; skew 500 ppm
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:47.808272 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:47.809085 3645 webserver.cc:460] Webserver started at http://127.0.98.2:40379/ using document root <none> and password file <none>
I20260620 04:35:47.809871 3645 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:47.809976 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:47.810230 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:47.812400 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "af9c1c751029483d9175175d27b34d9e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "07a225dff41fc57c53418a566478bb95"
server_key_iv: "c0afd11fe4bba31f9f4b258ce3e0bf3a"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.812950 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af9c1c751029483d9175175d27b34d9e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "07a225dff41fc57c53418a566478bb95"
server_key_iv: "c0afd11fe4bba31f9f4b258ce3e0bf3a"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.816632 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:47.819434 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.820849 3645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:47.820978 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "af9c1c751029483d9175175d27b34d9e"
format_stamp: "Formatted at 2026-06-20 04:35:47 on dist-test-slave-zmf4"
server_key: "07a225dff41fc57c53418a566478bb95"
server_key_iv: "c0afd11fe4bba31f9f4b258ce3e0bf3a"
server_key_version: "encryptionkey@0"
I20260620 04:35:47.821100 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:47.848974 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:47.852322 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:47.852537 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:47.853240 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:47.854174 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:47.854228 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.854270 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:47.854336 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:47.863072 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45639
I20260620 04:35:47.863138 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45639 every 8 connection(s)
I20260620 04:35:47.863973 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:47.871002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3645
I20260620 04:35:47.871151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:47.871443 392 external_mini_cluster.cc:1428] Setting key 2d880ff5de35ef56796ba07c4e5291bf
Jun 20 04:35:47 dist-test-slave-zmf4 krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:47.879236 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:47.865901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34783 (local address 127.0.98.62:46283)
0620 04:35:47.866033 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:47.866037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:47.866778 (+ 741us) server_negotiation.cc:407] Connection header received
0620 04:35:47.868337 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:47.868341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:47.868397 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:47.868501 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:47.869790 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.870314 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.871631 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.871910 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.872020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:47.874953 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:47.874956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:47.874959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:47.874982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:47.876602 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.877312 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.877315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.877317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.877390 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:47.877938 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:47.877941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:47.877942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:47.878136 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:47.878295 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:47.879018 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 04:35:47.879117 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:35:47.880515 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:47.866171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46283 (local address 127.0.98.2:34783)
0620 04:35:47.866620 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:47.866661 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:47.868069 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:47.868671 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:47.868678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:47.869018 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:47.869460 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:47.869630 (+ 170us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.870441 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.870444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:47.871498 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:47.871511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:47.872091 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:47.872096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:47.872310 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:47.872951 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:47.872972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:47.874763 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:47.876726 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:47.876736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:47.876756 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:47.877197 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:47.877577 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:47.877581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:47.877582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:47.877796 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:47.878316 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:47.878325 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:47.878691 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 04:35:47.878958 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 04:35:47.879337 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:35:47.881876 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46283
I20260620 04:35:47.882174 3774 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:47.882651 3774 heartbeater.cc:511] Master 127.0.98.62:46283 requested a full tablet report, sending...
I20260620 04:35:47.883811 3449 ts_manager.cc:194] Registered new tserver with Master: af9c1c751029483d9175175d27b34d9e (127.0.98.2:45639)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:47.884645 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:47.902727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:48.015442 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:48.015761 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:48.015811 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:48.020812 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:48.020881 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:48.020949 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:48.026957 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:48.028019 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:48.029150 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:48.039039 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:48.039193 3779 server_base.cc:1034] running on GCE node
W20260620 04:35:48.039052 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:48.039093 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:48.039791 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:48.040450 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:48.041656 3779 hybrid_clock.cc:648] HybridClock initialized: now 1781930148041635 us; error 32 us; skew 500 ppm
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:48.044433 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:48.045315 3779 webserver.cc:460] Webserver started at http://127.0.98.3:33241/ using document root <none> and password file <none>
I20260620 04:35:48.045882 3779 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:48.045972 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:48.046200 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:48.048257 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e3bc3fa219c4303bb7fe3f2e0b7a217"
format_stamp: "Formatted at 2026-06-20 04:35:48 on dist-test-slave-zmf4"
server_key: "2ca4f3b57214468bbf0060d9350b7beb"
server_key_iv: "e941412e3e513156c2c2a84b3b48033a"
server_key_version: "encryptionkey@0"
I20260620 04:35:48.048840 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e3bc3fa219c4303bb7fe3f2e0b7a217"
format_stamp: "Formatted at 2026-06-20 04:35:48 on dist-test-slave-zmf4"
server_key: "2ca4f3b57214468bbf0060d9350b7beb"
server_key_iv: "e941412e3e513156c2c2a84b3b48033a"
server_key_version: "encryptionkey@0"
I20260620 04:35:48.052377 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:35:48.054877 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:48.056176 3779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:48.056357 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "2e3bc3fa219c4303bb7fe3f2e0b7a217"
format_stamp: "Formatted at 2026-06-20 04:35:48 on dist-test-slave-zmf4"
server_key: "2ca4f3b57214468bbf0060d9350b7beb"
server_key_iv: "e941412e3e513156c2c2a84b3b48033a"
server_key_version: "encryptionkey@0"
I20260620 04:35:48.056447 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:48.069658 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:48.072770 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:48.072921 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:48.073410 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:48.074157 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:48.074209 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:48.074252 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:48.074267 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:48.083304 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41063
I20260620 04:35:48.083340 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41063 every 8 connection(s)
I20260620 04:35:48.084175 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:48.089432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3779
I20260620 04:35:48.089546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:48.090273 392 external_mini_cluster.cc:1428] Setting key 068ed99f583e6ca1952a4af31f2151c1
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:48.105230 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:48.086269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51677 (local address 127.0.98.62:46283)
0620 04:35:48.086449 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:48.086454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:48.087799 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 04:35:48.089955 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:48.089959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:48.090020 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:48.090104 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:48.092010 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.092835 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.094846 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.095253 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.095395 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:48.099895 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:48.099902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:48.099908 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:48.099943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:48.102275 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:48.102986 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:48.102992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:48.102994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:48.103165 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:48.103659 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:48.103662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:48.103664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:48.103944 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:48.104062 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:48.104828 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 04:35:48.105016 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 04:35:48.106194 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:48.086709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46283 (local address 127.0.98.3:51677)
0620 04:35:48.087515 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:48.087597 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:48.089712 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:48.090377 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:48.090389 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:48.090952 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:48.091688 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:48.091712 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.092999 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.093005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:48.094480 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:48.094491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.095439 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.095445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:48.095695 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:48.097016 (+ 1321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:48.097050 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:48.099609 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:48.102404 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:48.102410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:48.102435 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:48.102873 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:48.103293 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:48.103298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:48.103302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:48.103532 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:48.104096 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:48.104102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:48.104510 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 04:35:48.104752 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 04:35:48.105143 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":152,"threads_started":1}
I20260620 04:35:48.108116 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46283
I20260620 04:35:48.108397 3908 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:48.109350 3908 heartbeater.cc:511] Master 127.0.98.62:46283 requested a full tablet report, sending...
I20260620 04:35:48.112414 3449 ts_manager.cc:194] Registered new tserver with Master: 2e3bc3fa219c4303bb7fe3f2e0b7a217 (127.0.98.3:41063)
I20260620 04:35:48.113034 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51677
I20260620 04:35:48.119177 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930148, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930148, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:48.150002 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:48.140675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.0.98.62:46283)
0620 04:35:48.140822 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:48.140828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:48.140996 (+ 168us) server_negotiation.cc:407] Connection header received
0620 04:35:48.141181 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:48.141185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:48.141244 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:48.141335 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:48.142234 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.142747 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.143614 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:48.143825 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:48.143916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:48.146502 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:48.146505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:48.146507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:48.146530 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:48.148079 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:48.148644 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:48.148647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:48.148648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:48.148734 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:48.149066 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:48.149069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:48.149070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:48.149302 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:48.149411 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:48.149767 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 04:35:48.149869 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:35:48.150471 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34564
I20260620 04:35:48.151540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3511
I20260620 04:35:48.157555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3645
I20260620 04:35:48.163190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3779
I20260620 04:35:48.168392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3417
2026-06-20T04:35:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2179 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3920](info): set up 2 sockets
Jun 20 04:35:48 dist-test-slave-zmf4 krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260620 04:35:49.224438 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:49.349009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:49.457103 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:49.457347 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:49.457386 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:49.461328 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:49.461380 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:49.461398 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:49.461416 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:49.461431 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:49.466898 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:49.467943 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:49.469076 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:49.477756 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.477756 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.477792 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:49.477797 3933 server_base.cc:1034] running on GCE node
I20260620 04:35:49.478626 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:49.479741 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:49.480930 3933 hybrid_clock.cc:648] HybridClock initialized: now 1781930149480912 us; error 30 us; skew 500 ppm
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:49.483721 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:49.484665 3933 webserver.cc:460] Webserver started at http://127.0.98.62:33709/ using document root <none> and password file <none>
I20260620 04:35:49.485260 3933 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:49.485353 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:49.485594 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:49.487700 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "a9e5f80ed6324bc4815188ae917b0462"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "32009527b2c558d14058eba134d2971c"
server_key_iv: "a2894c6329fa41a00a5e090713a82ba8"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.488387 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9e5f80ed6324bc4815188ae917b0462"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "32009527b2c558d14058eba134d2971c"
server_key_iv: "a2894c6329fa41a00a5e090713a82ba8"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.492331 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:35:49.494786 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:49.496094 3933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:49.496282 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "a9e5f80ed6324bc4815188ae917b0462"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "32009527b2c558d14058eba134d2971c"
server_key_iv: "a2894c6329fa41a00a5e090713a82ba8"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.496398 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:49.510991 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:49.514245 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:49.514452 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:49.521605 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38185
I20260620 04:35:49.521737 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38185 every 8 connection(s)
I20260620 04:35:49.522562 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:49.525044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 3933
I20260620 04:35:49.525179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:49.525492 392 external_mini_cluster.cc:1428] Setting key 182abf0d98ef72fb6a72c18b1ef8bd36
I20260620 04:35:49.525561 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:49.531642 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462: Bootstrap starting.
I20260620 04:35:49.533841 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462: Neither blocks nor log segments found. Creating new log.
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:49.534543 4001 log.cc:824] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:49.536726 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462: No bootstrap required, opened a new log
I20260620 04:35:49.539806 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:49.526943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.98.62:38185)
0620 04:35:49.527399 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:49.527413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:49.527449 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:35:49.528292 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:49.528311 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:49.528619 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:49.528951 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:49.530026 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.531008 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.531908 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.532436 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.532612 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:49.535090 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:49.535099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:49.535118 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:49.535148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:49.536953 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:49.537421 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:49.537426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:49.537430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:49.537532 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:49.537825 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:49.537827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:49.537828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:49.538209 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:49.538350 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:49.538718 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 04:35:49.539061 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260620 04:35:49.540712 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260620 04:35:49.540951 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:49.541011 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e5f80ed6324bc4815188ae917b0462, State: Initialized, Role: FOLLOWER
I20260620 04:35:49.541442 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260620 04:35:49.541563 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:49.541602 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:49.541719 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:49.542596 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260620 04:35:49.542928 4001 leader_election.cc:302] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e5f80ed6324bc4815188ae917b0462; no voters:
I20260620 04:35:49.543246 4001 leader_election.cc:288] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:49.543418 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:49.543738 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [term 1 LEADER]: Becoming Leader. State: Replica: a9e5f80ed6324bc4815188ae917b0462, State: Running, Role: LEADER
I20260620 04:35:49.544047 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260620 04:35:49.544448 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:49.545789 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e5f80ed6324bc4815188ae917b0462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260620 04:35:49.545823 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e5f80ed6324bc4815188ae917b0462. Latest consensus state: current_term: 1 leader_uuid: "a9e5f80ed6324bc4815188ae917b0462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e5f80ed6324bc4815188ae917b0462" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260620 04:35:49.545902 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:49.545925 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:49.546244 4016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:49.549424 4016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:49.554948 4016 catalog_manager.cc:1347] Generated new cluster ID: 32c15ef1eb594a7babe7151790101681
I20260620 04:35:49.555039 4016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:49.565267 4016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:49.566161 4016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:49.571977 4016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e5f80ed6324bc4815188ae917b0462: Generated new TSK 0
I20260620 04:35:49.572674 4016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:49.607363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:49.722445 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:49.722698 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:49.722743 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:49.726857 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:49.726918 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:49.727006 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:49.732218 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:49.733225 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:49.734206 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:49.743472 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.743574 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.743448 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:49.744022 4027 server_base.cc:1034] running on GCE node
I20260620 04:35:49.744501 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:49.745168 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:49.746359 4027 hybrid_clock.cc:648] HybridClock initialized: now 1781930149746349 us; error 77 us; skew 500 ppm
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:49.749291 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:49.750149 4027 webserver.cc:460] Webserver started at http://127.0.98.1:40713/ using document root <none> and password file <none>
I20260620 04:35:49.750654 4027 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:49.750712 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:49.750871 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:49.752866 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b1d2bfbf22e427e8867bf4ff7433a0c"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "9b9f71fa495b69e1a32fdbedd436d82d"
server_key_iv: "5b640506bddf39d711d64846e2315147"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.753448 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1d2bfbf22e427e8867bf4ff7433a0c"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "9b9f71fa495b69e1a32fdbedd436d82d"
server_key_iv: "5b640506bddf39d711d64846e2315147"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.757077 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:35:49.759618 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:49.760970 4027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:35:49.761099 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "6b1d2bfbf22e427e8867bf4ff7433a0c"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "9b9f71fa495b69e1a32fdbedd436d82d"
server_key_iv: "5b640506bddf39d711d64846e2315147"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.761201 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:49.784018 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:49.787359 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:49.787575 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:49.788231 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:49.789055 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:49.789101 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:49.789141 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:49.789156 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:49.798612 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46141
I20260620 04:35:49.798635 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46141 every 8 connection(s)
I20260620 04:35:49.799638 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:49.803946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4027
I20260620 04:35:49.804080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:49.804425 392 external_mini_cluster.cc:1428] Setting key b1b55bd0637143cb8905f1c7fe1cf207
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:49.816185 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:49.801807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51519 (local address 127.0.98.62:38185)
0620 04:35:49.802039 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:49.802043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:49.802786 (+ 743us) server_negotiation.cc:407] Connection header received
0620 04:35:49.804523 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:49.804529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:49.804597 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:49.804676 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:49.806065 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.806749 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.808450 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.808886 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.809018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:49.811766 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:49.811771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:49.811775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:49.811806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:49.813967 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:49.814568 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:49.814570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:49.814572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:49.814660 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:49.815056 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:49.815059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:49.815060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:49.815251 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:49.815339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:49.815858 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 04:35:49.816018 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:35:49.817269 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:49.802163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.1:51519)
0620 04:35:49.802627 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:49.802670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:49.804215 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:49.804893 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:49.804900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:49.805321 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:49.805920 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:49.805935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.806897 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.806903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:49.808321 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:49.808331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:49.809064 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:49.809067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:49.809243 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:49.810134 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:49.810151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:49.811585 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:49.814127 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:49.814140 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:49.814155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:49.814433 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:49.814769 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:49.814771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:49.814773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:49.814933 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:49.815355 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:49.815361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:49.815609 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 04:35:49.815776 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 04:35:49.816264 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
I20260620 04:35:49.818657 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260620 04:35:49.818941 4156 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:49.819584 4156 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260620 04:35:49.821434 3965 ts_manager.cc:194] Registered new tserver with Master: 6b1d2bfbf22e427e8867bf4ff7433a0c (127.0.98.1:46141)
I20260620 04:35:49.822553 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:49.836041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:49.948758 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:49.949010 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:49.949055 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:49.953033 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:49.953087 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:49.953186 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:49.958349 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:49.959587 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:49.960646 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:49.970017 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.970017 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:49.970059 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:49.970410 4161 server_base.cc:1034] running on GCE node
I20260620 04:35:49.970854 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:49.971520 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:49.972726 4161 hybrid_clock.cc:648] HybridClock initialized: now 1781930149972700 us; error 39 us; skew 500 ppm
Jun 20 04:35:49 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:49.975522 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:49.976437 4161 webserver.cc:460] Webserver started at http://127.0.98.2:45725/ using document root <none> and password file <none>
I20260620 04:35:49.976996 4161 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:49.977108 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:49.977337 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:49.979315 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "15ae359a90374f4089d89e3d9eaac0d7"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "2c52d0982ef1adea1f03d5fec4ea6cff"
server_key_iv: "e574605a13e1a80640158e5d46b18f72"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.979875 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "15ae359a90374f4089d89e3d9eaac0d7"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "2c52d0982ef1adea1f03d5fec4ea6cff"
server_key_iv: "e574605a13e1a80640158e5d46b18f72"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.983562 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:35:49.986088 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:49.987375 4161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:49.987519 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "15ae359a90374f4089d89e3d9eaac0d7"
format_stamp: "Formatted at 2026-06-20 04:35:49 on dist-test-slave-zmf4"
server_key: "2c52d0982ef1adea1f03d5fec4ea6cff"
server_key_iv: "e574605a13e1a80640158e5d46b18f72"
server_key_version: "encryptionkey@0"
I20260620 04:35:49.987635 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:50.002725 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:50.006094 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:50.006307 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:50.006894 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:50.007694 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:50.007763 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:50.007834 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:50.007889 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:50.017786 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34503
I20260620 04:35:50.017822 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34503 every 8 connection(s)
I20260620 04:35:50.018638 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:50.022464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4161
I20260620 04:35:50.022588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:50.022902 392 external_mini_cluster.cc:1428] Setting key 0678fab204db87c03529ffd4eec046d5
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:50.035116 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.020460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42775 (local address 127.0.98.62:38185)
0620 04:35:50.020604 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:50.020609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:50.021466 (+ 857us) server_negotiation.cc:407] Connection header received
0620 04:35:50.023160 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:50.023163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:50.023210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:50.023276 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:50.024965 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.025818 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.027622 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.027856 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.027947 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.031124 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:50.031128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:50.031130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:50.031152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:50.032873 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.033385 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.033389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.033391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.033504 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.033900 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.033903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.033904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.034111 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:50.034198 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:50.034805 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 04:35:50.034923 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:35:50.036091 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.020853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.2:42775)
0620 04:35:50.021320 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:50.021362 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:50.022911 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:50.023634 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:50.023643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:50.024097 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:50.024813 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:50.024828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.025956 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.025959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:50.027471 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:50.027485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.028015 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.028022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:50.028311 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.028993 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:50.029018 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:50.030932 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:50.032992 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:50.032998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:50.033013 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:50.033279 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:50.033637 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:50.033640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:50.033641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:50.033790 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:50.034214 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:50.034219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:50.034507 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 04:35:50.034678 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:35:50.035070 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260620 04:35:50.037648 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260620 04:35:50.037935 4290 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:50.038508 4290 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260620 04:35:50.039801 3965 ts_manager.cc:194] Registered new tserver with Master: 15ae359a90374f4089d89e3d9eaac0d7 (127.0.98.2:34503)
I20260620 04:35:50.040710 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:50.052536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:50.167954 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:50.168255 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:50.168344 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:50.172377 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:50.172452 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:50.172544 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:50.177758 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:50.178781 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:50.179761 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:50.189374 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:50.189603 4295 server_base.cc:1034] running on GCE node
W20260620 04:35:50.189374 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:50.189395 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:50.190155 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:50.190835 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:50.192092 4295 hybrid_clock.cc:648] HybridClock initialized: now 1781930150192062 us; error 42 us; skew 500 ppm
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:50.194796 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:50.195674 4295 webserver.cc:460] Webserver started at http://127.0.98.3:38275/ using document root <none> and password file <none>
I20260620 04:35:50.196180 4295 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:50.196300 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:50.196468 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:50.198414 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bcafaed309549b28b3342f91c75d28b"
format_stamp: "Formatted at 2026-06-20 04:35:50 on dist-test-slave-zmf4"
server_key: "3e141eb04349ed47d90e7be30b296f37"
server_key_iv: "cc095a12036f0eb8efe0ae1ebafee3cb"
server_key_version: "encryptionkey@0"
I20260620 04:35:50.198963 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bcafaed309549b28b3342f91c75d28b"
format_stamp: "Formatted at 2026-06-20 04:35:50 on dist-test-slave-zmf4"
server_key: "3e141eb04349ed47d90e7be30b296f37"
server_key_iv: "cc095a12036f0eb8efe0ae1ebafee3cb"
server_key_version: "encryptionkey@0"
I20260620 04:35:50.203459 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 04:35:50.206135 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:50.207475 4295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:35:50.207576 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "8bcafaed309549b28b3342f91c75d28b"
format_stamp: "Formatted at 2026-06-20 04:35:50 on dist-test-slave-zmf4"
server_key: "3e141eb04349ed47d90e7be30b296f37"
server_key_iv: "cc095a12036f0eb8efe0ae1ebafee3cb"
server_key_version: "encryptionkey@0"
I20260620 04:35:50.207662 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:50.225303 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:50.228466 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:50.228608 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:50.229107 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:50.229899 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:50.229944 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:50.229982 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:50.229997 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:50.239162 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34781
I20260620 04:35:50.239190 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34781 every 8 connection(s)
I20260620 04:35:50.240056 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:50.249739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4295
I20260620 04:35:50.249869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:50.250226 392 external_mini_cluster.cc:1428] Setting key 143e349a6963c76df32451c92103451d
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:50.254846 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.241927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39581 (local address 127.0.98.62:38185)
0620 04:35:50.242080 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:50.242084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:50.243203 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 04:35:50.245195 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:50.245199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:50.245264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:50.245371 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:50.246581 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.247126 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.248270 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.248492 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.248611 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.251366 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:50.251370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:50.251373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:50.251404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:50.252979 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.253462 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.253465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.253467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.253554 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.253908 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.253910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.253911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.254073 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:50.254162 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:50.254626 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 04:35:50.254732 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:35:50.255602 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.242279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38185 (local address 127.0.98.3:39581)
0620 04:35:50.242946 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:50.243017 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:50.244930 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:50.245545 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:50.245552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:50.245935 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:50.246434 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:50.246448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.247254 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.247260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:50.248121 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:50.248130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.248639 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.248644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:50.248810 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.249367 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:50.249382 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:50.251211 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:50.253080 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:50.253085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:50.253096 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:50.253345 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:50.253678 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:50.253681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:50.253682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:50.253821 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:50.254163 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:50.254168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:50.254386 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 04:35:50.254542 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 04:35:50.254794 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":181,"threads_started":1}
I20260620 04:35:50.256933 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38185
I20260620 04:35:50.257196 4424 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:50.257664 4424 heartbeater.cc:511] Master 127.0.98.62:38185 requested a full tablet report, sending...
I20260620 04:35:50.258708 3965 ts_manager.cc:194] Registered new tserver with Master: 8bcafaed309549b28b3342f91c75d28b (127.0.98.3:34781)
I20260620 04:35:50.259339 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39581
I20260620 04:35:50.264434 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:50.290657 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.281045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.98.1:46141)
0620 04:35:50.281330 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:50.281350 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:50.281383 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:35:50.281436 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:50.281441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:50.281637 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:50.281766 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:50.282461 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.283251 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.284122 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.284557 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.284650 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.287152 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:50.287160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:50.287173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:50.287197 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:50.288874 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.289317 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.289322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.289323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.289402 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.289672 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.289679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.289683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.289916 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:50.290145 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:50.290298 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:35:50.290500 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:50.302168 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:50.293775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.0.98.62:38185)
0620 04:35:50.293990 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:50.293995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:50.294124 (+ 129us) server_negotiation.cc:407] Connection header received
0620 04:35:50.294274 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:50.294278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:50.294330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:50.294416 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:50.294991 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.295499 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.296300 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:50.296519 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:50.296645 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:50.299018 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:50.299023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:50.299030 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:50.299053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:50.300532 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.300929 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.300933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.300935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.301056 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:50.301318 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:50.301320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:50.301321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:50.301531 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:50.301646 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:50.301894 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 04:35:50.301982 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 04:35:50.302618 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34046
I20260620 04:35:50.303247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4027
I20260620 04:35:50.308724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4161
I20260620 04:35:50.314246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4295
I20260620 04:35:50.319589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 3933
2026-06-20T04:35:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2149 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[4436](info): set up 2 sockets
Jun 20 04:35:50 dist-test-slave-zmf4 krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260620 04:35:51.376765 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/test-user.keytab.
W20260620 04:35:51.854166 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.348s user 0.000s sys 0.002s
Jun 20 04:35:51 dist-test-slave-zmf4 krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:51.939901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:52.054201 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:52.054593 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:52.054658 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:52.058902 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:52.058969 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:52.058990 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:52.059005 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:52.059023 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:52.064409 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:52.065527 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:52.066568 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:52.075421 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.075642 4449 server_base.cc:1034] running on GCE node
W20260620 04:35:52.075419 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.075419 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.076349 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:52.077479 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:52.078732 4449 hybrid_clock.cc:648] HybridClock initialized: now 1781930152078706 us; error 35 us; skew 500 ppm
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:52.085686 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:52.086472 4449 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:52.086571 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:52.086807 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:52.088933 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "1d8731fe8d444c87a43601919621038e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "1a4d36a0bef08f78ca66ebe4ac3dc4f4"
server_key_iv: "33fdc682434c9d2fe177247cb7a85efd"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.089504 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d8731fe8d444c87a43601919621038e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "1a4d36a0bef08f78ca66ebe4ac3dc4f4"
server_key_iv: "33fdc682434c9d2fe177247cb7a85efd"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.093238 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:52.096491 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.098067 4449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:52.098254 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "1d8731fe8d444c87a43601919621038e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "1a4d36a0bef08f78ca66ebe4ac3dc4f4"
server_key_iv: "33fdc682434c9d2fe177247cb7a85efd"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.098379 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:52.117157 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:52.120476 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:52.120711 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:52.127950 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42903
I20260620 04:35:52.128295 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42903 every 8 connection(s)
I20260620 04:35:52.128667 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:52.131449 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:52.136760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4449
I20260620 04:35:52.136896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:52.137210 392 external_mini_cluster.cc:1428] Setting key 30671c8a94daa552e04cc1ce8617eede
I20260620 04:35:52.138480 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: Bootstrap starting.
I20260620 04:35:52.140818 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:52.141775 4516 log.cc:824] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:52.143914 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: No bootstrap required, opened a new log
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:52.148528 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260620 04:35:52.148759 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:52.148794 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8731fe8d444c87a43601919621038e, State: Initialized, Role: FOLLOWER
I20260620 04:35:52.149302 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260620 04:35:52.149423 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:52.149507 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:52.149612 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:52.150715 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260620 04:35:52.151118 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8731fe8d444c87a43601919621038e; no voters:
I20260620 04:35:52.151466 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:52.151546 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:52.151757 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [term 1 LEADER]: Becoming Leader. State: Replica: 1d8731fe8d444c87a43601919621038e, State: Running, Role: LEADER
I20260620 04:35:52.152104 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } }
I20260620 04:35:52.152808 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:52.153312 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8731fe8d444c87a43601919621038e. Latest consensus state: current_term: 1 leader_uuid: "1d8731fe8d444c87a43601919621038e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } } }
I20260620 04:35:52.153429 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:52.154048 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8731fe8d444c87a43601919621038e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8731fe8d444c87a43601919621038e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42903 } } }
I20260620 04:35:52.154162 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:52.154484 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.138770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.98.62:42903)
0620 04:35:52.139219 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.139238 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.139274 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:35:52.139982 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.140000 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.140495 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.140807 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:52.141730 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.142747 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.143911 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.144519 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.144652 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.147687 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:52.147693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:52.147708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:52.147745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:52.149989 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.150545 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.150550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.150555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.150677 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.150995 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.150999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.151001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.151479 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:52.152666 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.152793 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.153110 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":148,"threads_started":1}
W20260620 04:35:52.157948 4536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:35:52.158020 4536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:35:52.158099 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:52.159089 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:52.165082 4527 catalog_manager.cc:1347] Generated new cluster ID: c11d1ef6606e44c1852e0afc801ea1de
I20260620 04:35:52.165163 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:52.180682 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:52.181926 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:52.190362 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: Generated new TSK 0
I20260620 04:35:52.191201 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:52.220615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:52.339157 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:52.339421 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:52.339466 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:52.343686 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:52.343758 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:52.343819 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:52.349565 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:52.350580 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:52.351547 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:52.361387 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.361408 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.361446 4542 server_base.cc:1034] running on GCE node
W20260620 04:35:52.361408 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.362080 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:52.362715 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:52.363894 4542 hybrid_clock.cc:648] HybridClock initialized: now 1781930152363856 us; error 51 us; skew 500 ppm
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:52.366912 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:52.367643 4542 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:52.367745 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:52.367964 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:52.370023 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "6e24a8f9725e40c5b57baf8a70cb3a49"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "49cdb1a25911fb923074c0f09ab0bd0a"
server_key_iv: "132fff59e59069eba6ee8b42d6e07b55"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.370581 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6e24a8f9725e40c5b57baf8a70cb3a49"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "49cdb1a25911fb923074c0f09ab0bd0a"
server_key_iv: "132fff59e59069eba6ee8b42d6e07b55"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.374226 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:35:52.376869 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.378392 4542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:52.378530 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "6e24a8f9725e40c5b57baf8a70cb3a49"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "49cdb1a25911fb923074c0f09ab0bd0a"
server_key_iv: "132fff59e59069eba6ee8b42d6e07b55"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.378647 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:52.400492 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:52.403831 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:52.404042 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:52.404678 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:52.405586 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:52.405658 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.405723 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:52.405769 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.415481 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34095
I20260620 04:35:52.415500 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34095 every 8 connection(s)
I20260620 04:35:52.416652 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:52.417182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4542
I20260620 04:35:52.417320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:52.417616 392 external_mini_cluster.cc:1428] Setting key 63e79b88733bd1b81a5eeadab09a9720
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:52.434878 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.419338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41413 (local address 127.0.98.62:42903)
0620 04:35:52.419491 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.419496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.420571 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 04:35:52.421973 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.421980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.422041 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.422140 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:52.423426 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.424417 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.425985 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.426417 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.426558 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.430736 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:52.430743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:52.430749 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:52.430783 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:52.432455 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.433095 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.433098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.433099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.433168 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.433642 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.433648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.433650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.433858 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:52.433944 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.434650 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.434755 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:35:52.436152 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.419866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.1:41413)
0620 04:35:52.420400 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.420455 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.421764 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.422338 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.422344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.422704 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:52.423272 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.423286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.424663 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.424668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.425801 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.425817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.426656 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.426660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.426919 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.427498 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:52.427515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:52.430548 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:52.432576 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.432585 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.432605 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.432992 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.433272 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.433278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.433282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.433477 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.433977 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:52.433984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:52.434338 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.434598 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.435000 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":146,"threads_started":1}
I20260620 04:35:52.437866 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260620 04:35:52.438125 4670 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:52.438575 4670 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260620 04:35:52.439981 4477 ts_manager.cc:194] Registered new tserver with Master: 6e24a8f9725e40c5b57baf8a70cb3a49 (127.0.98.1:34095)
I20260620 04:35:52.441139 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:52.447304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:52.564579 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:52.564855 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:52.564929 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:52.568967 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:52.569046 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:52.569139 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:52.574353 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:52.575383 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:52.576449 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:52.586164 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.586177 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.586215 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.586532 4675 server_base.cc:1034] running on GCE node
I20260620 04:35:52.586938 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:52.587541 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:52.588778 4675 hybrid_clock.cc:648] HybridClock initialized: now 1781930152588736 us; error 52 us; skew 500 ppm
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:52.591691 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:52.592397 4675 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:52.592460 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:52.592623 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:52.594565 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "f41e28e11b1e474f865836cbd3def90e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "2739b6a5a418244fff7f88dc62631f1e"
server_key_iv: "6c7d3186507f605208f00db6d1eca2f5"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.595069 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f41e28e11b1e474f865836cbd3def90e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "2739b6a5a418244fff7f88dc62631f1e"
server_key_iv: "6c7d3186507f605208f00db6d1eca2f5"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.598628 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:35:52.601073 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.602612 4675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:52.602706 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "f41e28e11b1e474f865836cbd3def90e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "2739b6a5a418244fff7f88dc62631f1e"
server_key_iv: "6c7d3186507f605208f00db6d1eca2f5"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.602789 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:52.620486 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:52.623617 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:52.623775 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:52.624315 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:52.625200 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:52.625247 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.625319 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:52.625355 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.634388 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43609
I20260620 04:35:52.634454 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43609 every 8 connection(s)
I20260620 04:35:52.634969 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:52.635182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4675
I20260620 04:35:52.635334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:52.635663 392 external_mini_cluster.cc:1428] Setting key 0d139c8f8e320e65d555a2f648493534
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:52.654249 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.636994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40415 (local address 127.0.98.62:42903)
0620 04:35:52.637256 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.637260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.638170 (+ 910us) server_negotiation.cc:407] Connection header received
0620 04:35:52.639956 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.639959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.640012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.640092 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:52.641736 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.642263 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.644122 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.644427 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.644519 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.649762 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:52.649769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:52.649776 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:52.649812 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:52.651600 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.652322 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.652325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.652331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.652457 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.652904 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.652907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.652909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.653185 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:52.653350 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.653912 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.654042 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260620 04:35:52.655139 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.637495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.2:40415)
0620 04:35:52.638006 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.638054 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.639694 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.640323 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.640334 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.640868 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:52.641545 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.641572 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.642404 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.642410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.643973 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.643987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.644596 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.644603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.644819 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.645442 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:52.645469 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:52.649583 (+ 4114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:52.651722 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.651731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.651749 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.652131 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.652584 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.652587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.652589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.652782 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.653356 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:52.653365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:52.653642 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.653856 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.654220 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
I20260620 04:35:52.656504 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260620 04:35:52.656752 4803 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:52.657280 4803 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260620 04:35:52.658499 4477 ts_manager.cc:194] Registered new tserver with Master: f41e28e11b1e474f865836cbd3def90e (127.0.98.2:43609)
I20260620 04:35:52.659272 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:52.665010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:52.771996 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:52.772255 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:52.772298 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:52.776226 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:52.776281 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:52.776376 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:52.781461 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:52.782471 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:52.783490 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:52.792469 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.792455 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:52.792459 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:52.792693 4808 server_base.cc:1034] running on GCE node
I20260620 04:35:52.793159 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:52.793769 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:52.794973 4808 hybrid_clock.cc:648] HybridClock initialized: now 1781930152794953 us; error 32 us; skew 500 ppm
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:52.797978 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:52.798674 4808 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:52.798766 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:52.798997 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:52.801052 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "ecdfff9083c7440cbae1c530cfcf661e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "e11a3f83f5540747968cbff41b0056f4"
server_key_iv: "b1f319813c4b93a9219993752c0bbcab"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.801630 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ecdfff9083c7440cbae1c530cfcf661e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "e11a3f83f5540747968cbff41b0056f4"
server_key_iv: "b1f319813c4b93a9219993752c0bbcab"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.805349 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:52.807955 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.809371 4808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:52.809509 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "ecdfff9083c7440cbae1c530cfcf661e"
format_stamp: "Formatted at 2026-06-20 04:35:52 on dist-test-slave-zmf4"
server_key: "e11a3f83f5540747968cbff41b0056f4"
server_key_iv: "b1f319813c4b93a9219993752c0bbcab"
server_key_version: "encryptionkey@0"
I20260620 04:35:52.809623 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:52.823518 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:52.826714 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:52.826910 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:52.827462 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:52.828322 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:52.828389 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.828454 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:52.828505 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:52.837716 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36369
I20260620 04:35:52.837777 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36369 every 8 connection(s)
I20260620 04:35:52.838299 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:52.840921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4808
I20260620 04:35:52.841054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:52.841360 392 external_mini_cluster.cc:1428] Setting key cb3015a9df7e2d6dbca695de312a7cde
Jun 20 04:35:52 dist-test-slave-zmf4 krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:52.851989 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.840037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34385 (local address 127.0.98.62:42903)
0620 04:35:52.840289 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.840293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.841120 (+ 827us) server_negotiation.cc:407] Connection header received
0620 04:35:52.842828 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.842831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.842886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.842989 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:52.844234 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.844747 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.845862 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.846082 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.846195 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.848624 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:52.848628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:52.848630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:52.848653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:52.849968 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.850550 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.850555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.850557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.850640 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.850969 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.850974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.850976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.851148 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:52.851255 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.851710 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.851822 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 04:35:52.852857 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.840385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42903 (local address 127.0.98.3:34385)
0620 04:35:52.840945 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.841018 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.842609 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.843165 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.843171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.843615 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:52.844060 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.844073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.844873 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.844876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.845737 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.845746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.846226 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.846229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.846399 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.846958 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:52.846976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:52.848462 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:52.850186 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.850191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.850205 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.850448 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.850739 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:52.850742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:52.850744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:52.850878 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:52.851253 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:52.851259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:52.851486 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.851665 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.851962 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":105,"threads_started":1}
I20260620 04:35:52.854151 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42903
I20260620 04:35:52.854389 4936 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:52.854938 4936 heartbeater.cc:511] Master 127.0.98.62:42903 requested a full tablet report, sending...
I20260620 04:35:52.855926 4477 ts_manager.cc:194] Registered new tserver with Master: ecdfff9083c7440cbae1c530cfcf661e (127.0.98.3:36369)
I20260620 04:35:52.856528 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34385
I20260620 04:35:52.866971 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:35:52.876307 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.868940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.0.98.62:42903)
0620 04:35:52.869063 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.869066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.869139 (+ 73us) server_negotiation.cc:407] Connection header received
0620 04:35:52.869285 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.869287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.869328 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.869408 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:52.870169 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.870718 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.871696 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.871912 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.871998 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.873463 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:52.873466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:52.873468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:52.873488 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:52.874661 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.875072 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.875075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.875076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.875161 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:52.875472 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:52.875475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:52.875476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:52.875697 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:52.875810 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.875983 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.876108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:35:52.879058 4477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:52.882102 4477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:52.895985 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.890981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.98.3:36369)
0620 04:35:52.891463 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.891473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.891491 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:35:52.891560 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.891565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.891744 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.891894 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:52.892230 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.892998 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.894504 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895294 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895402 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895537 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.895734 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.895824 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":174,"threads_started":1}
I20260620 04:35:52.895993 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.890033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36369 (local address 127.0.0.1:45868)
0620 04:35:52.890451 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.890465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.890552 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.891890 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.891896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.891917 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:52.892083 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.892092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.893139 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.893143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.894362 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.894371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895425 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.895528 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895563 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.895703 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.895820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260620 04:35:52.896047 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.890692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34095 (local address 127.0.0.1:53146)
0620 04:35:52.891154 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.891170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.891255 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.892063 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.892065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.892080 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:52.892185 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.892195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.893615 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.893618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.894684 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.894692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895618 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.895714 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895736 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.895802 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.895874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260620 04:35:52.896243 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.889993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43609 (local address 127.0.0.1:57276)
0620 04:35:52.890382 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:52.890424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:52.890539 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:52.891981 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:52.891983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:52.891995 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:52.892108 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.892116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.893576 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.893580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.894743 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:52.894751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895875 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:52.895960 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895980 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:35:52.896043 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:35:52.896109 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":55,"thread_start_us":99,"threads_started":1}
I20260620 04:35:52.896552 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.890969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.0.98.1:34095)
0620 04:35:52.891462 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.891470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.891491 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:35:52.891561 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.891565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.891744 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.891894 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:52.892332 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.893452 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.894803 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895328 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895454 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895595 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.896217 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.896393 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":103,"threads_started":1}
I20260620 04:35:52.896797 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:52.890805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.98.2:43609)
0620 04:35:52.891332 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:52.891345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:52.891368 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:35:52.891442 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:52.891447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:52.891672 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:52.891820 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:52.892244 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.893426 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.894870 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:52.895595 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:52.895701 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:52.895822 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:52.896566 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 04:35:52.896671 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":138,"threads_started":1}
I20260620 04:35:52.899046 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec (DEFAULT_TABLE table=test-table [id=d20e08310c154ebda090732ff786cd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:52.899187 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec (DEFAULT_TABLE table=test-table [id=d20e08310c154ebda090732ff786cd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:52.899189 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec (DEFAULT_TABLE table=test-table [id=d20e08310c154ebda090732ff786cd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:52.899904 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d774c8a9cfa4981a59fb8395587d4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:52.900267 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d774c8a9cfa4981a59fb8395587d4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:52.900262 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d774c8a9cfa4981a59fb8395587d4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:52.906431 4953 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: Bootstrap starting.
I20260620 04:35:52.907538 4955 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Bootstrap starting.
I20260620 04:35:52.907629 4954 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Bootstrap starting.
I20260620 04:35:52.908875 4953 tablet_bootstrap.cc:654] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:52.909622 4955 tablet_bootstrap.cc:654] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:52.909622 4954 tablet_bootstrap.cc:654] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:52.910163 4953 log.cc:824] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:52.910979 4954 log.cc:824] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:52.910979 4955 log.cc:824] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:52.912410 4953 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: No bootstrap required, opened a new log
I20260620 04:35:52.912612 4953 ts_tablet_manager.cc:1397] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:35:52.912909 4954 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: No bootstrap required, opened a new log
I20260620 04:35:52.912909 4955 tablet_bootstrap.cc:492] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: No bootstrap required, opened a new log
I20260620 04:35:52.913076 4954 ts_tablet_manager.cc:1397] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:35:52.913076 4955 ts_tablet_manager.cc:1397] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:35:52.915951 4953 raft_consensus.cc:348] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.916208 4953 raft_consensus.cc:374] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:52.916304 4953 raft_consensus.cc:729] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e24a8f9725e40c5b57baf8a70cb3a49, State: Initialized, Role: FOLLOWER
I20260620 04:35:52.916769 4953 consensus_queue.cc:260] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.916832 4954 raft_consensus.cc:348] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.916832 4955 raft_consensus.cc:348] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.917111 4954 raft_consensus.cc:374] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:52.917111 4955 raft_consensus.cc:374] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:52.917172 4955 raft_consensus.cc:729] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41e28e11b1e474f865836cbd3def90e, State: Initialized, Role: FOLLOWER
I20260620 04:35:52.917171 4954 raft_consensus.cc:729] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdfff9083c7440cbae1c530cfcf661e, State: Initialized, Role: FOLLOWER
I20260620 04:35:52.917588 4670 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260620 04:35:52.917709 4955 consensus_queue.cc:260] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.917709 4954 consensus_queue.cc:260] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:52.918542 4936 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260620 04:35:52.918601 4803 heartbeater.cc:503] Master 127.0.98.62:42903 was elected leader, sending a full tablet report...
I20260620 04:35:52.918701 4955 ts_tablet_manager.cc:1428] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 04:35:52.919831 4954 ts_tablet_manager.cc:1428] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 04:35:52.919854 4953 ts_tablet_manager.cc:1428] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:35:53.081751 4961 raft_consensus.cc:482] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:53.081944 4961 raft_consensus.cc:504] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:53.083093 4961 leader_election.cc:288] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdfff9083c7440cbae1c530cfcf661e (127.0.98.3:36369), 6e24a8f9725e40c5b57baf8a70cb3a49 (127.0.98.1:34095)
I20260620 04:35:53.087005 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.083542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45435 (local address 127.0.98.3:36369)
0620 04:35:53.083668 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.083673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.083689 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:35:53.083751 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.083754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.083795 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.083897 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:53.084272 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.084877 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.085988 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.086427 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.086505 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:53.086595 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.086799 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.086871 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:35:53.087014 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.083805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51585 (local address 127.0.98.1:34095)
0620 04:35:53.083936 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.083941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.084071 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:35:53.084136 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.084139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.084228 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.084304 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:35:53.084533 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.085042 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.086072 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.086450 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.086572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:53.086631 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.086831 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.086900 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:35:53.087014 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.083381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36369 (local address 127.0.98.2:45435)
0620 04:35:53.083511 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:53.083527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:53.083621 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:53.083949 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:53.083955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:53.083976 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:53.084122 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:53.084129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.084986 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.084989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:53.085867 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:53.085876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.086550 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.086553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:53.086672 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:53.086700 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:35:53.086759 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:35:53.086883 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":46}
I20260620 04:35:53.087028 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.083668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34095 (local address 127.0.98.2:51585)
0620 04:35:53.083966 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:53.083982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:53.084074 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:53.084311 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:53.084314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:53.084330 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:35:53.084430 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:53.084437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.085150 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.085152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:53.085953 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:53.085964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.086596 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.086610 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:53.086694 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:53.086710 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:35:53.086800 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 04:35:53.086882 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260620 04:35:53.087637 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d774c8a9cfa4981a59fb8395587d4ec" candidate_uuid: "f41e28e11b1e474f865836cbd3def90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" is_pre_election: true
I20260620 04:35:53.087675 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d774c8a9cfa4981a59fb8395587d4ec" candidate_uuid: "f41e28e11b1e474f865836cbd3def90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdfff9083c7440cbae1c530cfcf661e" is_pre_election: true
I20260620 04:35:53.087924 4624 raft_consensus.cc:2454] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41e28e11b1e474f865836cbd3def90e in term 0.
I20260620 04:35:53.087942 4890 raft_consensus.cc:2454] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41e28e11b1e474f865836cbd3def90e in term 0.
I20260620 04:35:53.088380 4690 leader_election.cc:302] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e24a8f9725e40c5b57baf8a70cb3a49, f41e28e11b1e474f865836cbd3def90e; no voters:
I20260620 04:35:53.088647 4961 raft_consensus.cc:2781] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:53.088718 4961 raft_consensus.cc:482] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:53.088748 4961 raft_consensus.cc:3037] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 0 FOLLOWER]: Advancing to term 1
W20260620 04:35:53.089095 4937 tablet.cc:2369] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:53.089983 4961 raft_consensus.cc:504] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:53.090343 4961 leader_election.cc:288] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [CANDIDATE]: Term 1 election: Requested vote from peers ecdfff9083c7440cbae1c530cfcf661e (127.0.98.3:36369), 6e24a8f9725e40c5b57baf8a70cb3a49 (127.0.98.1:34095)
I20260620 04:35:53.090786 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d774c8a9cfa4981a59fb8395587d4ec" candidate_uuid: "f41e28e11b1e474f865836cbd3def90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdfff9083c7440cbae1c530cfcf661e"
I20260620 04:35:53.090939 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d774c8a9cfa4981a59fb8395587d4ec" candidate_uuid: "f41e28e11b1e474f865836cbd3def90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49"
I20260620 04:35:53.090898 4890 raft_consensus.cc:3037] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:53.091100 4624 raft_consensus.cc:3037] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:53.092110 4624 raft_consensus.cc:2454] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e28e11b1e474f865836cbd3def90e in term 1.
I20260620 04:35:53.092773 4690 leader_election.cc:302] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e24a8f9725e40c5b57baf8a70cb3a49, f41e28e11b1e474f865836cbd3def90e; no voters:
I20260620 04:35:53.092908 4890 raft_consensus.cc:2454] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e28e11b1e474f865836cbd3def90e in term 1.
I20260620 04:35:53.092948 4961 raft_consensus.cc:2781] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:53.093240 4961 raft_consensus.cc:686] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 1 LEADER]: Becoming Leader. State: Replica: f41e28e11b1e474f865836cbd3def90e, State: Running, Role: LEADER
I20260620 04:35:53.093875 4961 consensus_queue.cc:237] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } }
I20260620 04:35:53.097168 4480 catalog_manager.cc:5495] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e reported cstate change: term changed from 0 to 1, leader changed from <none> to f41e28e11b1e474f865836cbd3def90e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f41e28e11b1e474f865836cbd3def90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e28e11b1e474f865836cbd3def90e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20260620 04:35:53.133054 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.128983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.0.98.2:43609)
0620 04:35:53.129121 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.129125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.129141 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:35:53.129272 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.129275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.129335 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.129438 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:35:53.129653 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.130385 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.131299 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:53.131623 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:53.131733 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:53.132048 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:35:53.132566 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:35:53.132675 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.132816 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.132878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 04:35:53.136739 4804 tablet.cc:2369] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:53.139109 4624 raft_consensus.cc:1261] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 1 FOLLOWER]: Refusing update from remote peer f41e28e11b1e474f865836cbd3def90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:53.139210 4890 raft_consensus.cc:1261] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 1 FOLLOWER]: Refusing update from remote peer f41e28e11b1e474f865836cbd3def90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:53.139804 4961 consensus_queue.cc:1035] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e24a8f9725e40c5b57baf8a70cb3a49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:53.140004 4963 consensus_queue.cc:1035] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdfff9083c7440cbae1c530cfcf661e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:53.147727 4967 mvcc.cc:205] Tried to move back new op lower bound from 7298785907250913280 to 7298785907074711552. Current Snapshot: MvccSnapshot[applied={T|T < 7298785907250913280}]
I20260620 04:35:53.147877 4968 mvcc.cc:205] Tried to move back new op lower bound from 7298785907250913280 to 7298785907074711552. Current Snapshot: MvccSnapshot[applied={T|T < 7298785907250913280}]
I20260620 04:35:53.156841 4480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:53.157047 4480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:53.159734 4480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d8731fe8d444c87a43601919621038e: Sending DeleteTablet for 3 replicas of tablet 5d774c8a9cfa4981a59fb8395587d4ec
I20260620 04:35:53.160527 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57276
I20260620 04:35:53.160645 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53146
I20260620 04:35:53.160722 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d774c8a9cfa4981a59fb8395587d4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260620 04:35:53.161043 4977 tablet_replica.cc:330] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49: stopping tablet replica
I20260620 04:35:53.161276 4978 tablet_replica.cc:330] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: stopping tablet replica
I20260620 04:35:53.161382 4977 raft_consensus.cc:2229] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:53.161425 4976 tablet_replica.cc:330] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: stopping tablet replica
I20260620 04:35:53.161722 4977 raft_consensus.cc:2258] T 5d774c8a9cfa4981a59fb8395587d4ec P 6e24a8f9725e40c5b57baf8a70cb3a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:53.161957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4542
I20260620 04:35:53.163249 4976 raft_consensus.cc:2229] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:53.163619 4976 raft_consensus.cc:2258] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:53.163851 4978 raft_consensus.cc:2229] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:53.164134 4978 raft_consensus.cc:2258] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:53.165608 4976 ts_tablet_manager.cc:1905] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:53.165608 4978 ts_tablet_manager.cc:1905] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:53.168301 4978 ts_tablet_manager.cc:1918] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:53.168301 4976 ts_tablet_manager.cc:1918] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:53.168398 4976 log.cc:1196] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/5d774c8a9cfa4981a59fb8395587d4ec
I20260620 04:35:53.168509 4978 log.cc:1196] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/5d774c8a9cfa4981a59fb8395587d4ec
I20260620 04:35:53.168737 4976 ts_tablet_manager.cc:1939] T 5d774c8a9cfa4981a59fb8395587d4ec P f41e28e11b1e474f865836cbd3def90e: Deleting consensus metadata
I20260620 04:35:53.168946 4978 ts_tablet_manager.cc:1939] T 5d774c8a9cfa4981a59fb8395587d4ec P ecdfff9083c7440cbae1c530cfcf661e: Deleting consensus metadata
I20260620 04:35:53.170137 4467 catalog_manager.cc:4841] TS f41e28e11b1e474f865836cbd3def90e (127.0.98.2:43609): tablet 5d774c8a9cfa4981a59fb8395587d4ec (table test-table [id=d20e08310c154ebda090732ff786cd3f]) successfully deleted
I20260620 04:35:53.170140 4464 catalog_manager.cc:4841] TS ecdfff9083c7440cbae1c530cfcf661e (127.0.98.3:36369): tablet 5d774c8a9cfa4981a59fb8395587d4ec (table test-table [id=d20e08310c154ebda090732ff786cd3f]) successfully deleted
W20260620 04:35:53.171339 4464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34095 (error 108)
I20260620 04:35:53.172453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4675
I20260620 04:35:53.172649 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.172050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34095 (local address 127.0.0.1:53156)
0620 04:35:53.172367 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:53.172441 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
W20260620 04:35:53.173504 4464 catalog_manager.cc:4584] TS 6e24a8f9725e40c5b57baf8a70cb3a49 (127.0.98.1:34095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d774c8a9cfa4981a59fb8395587d4ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34095: connect: Connection refused (error 111)
I20260620 04:35:53.178263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4808
I20260620 04:35:53.183856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4449
2026-06-20T04:35:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2865 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T04:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:53Z Disabled control of system clock
I20260620 04:35:53.214793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36355
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:35:53.326670 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:53.326941 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:53.326982 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:53.331248 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:53.331313 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:53.331329 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:53.331346 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:53.331362 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:53.337195 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:53.338338 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:53.339721 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:53.348405 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:53.348461 4983 server_base.cc:1034] running on GCE node
W20260620 04:35:53.348405 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.348453 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:53.349225 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:53.350277 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:53.351459 4983 hybrid_clock.cc:648] HybridClock initialized: now 1781930153351447 us; error 25 us; skew 500 ppm
I20260620 04:35:53.353544 4983 webserver.cc:460] Webserver started at http://127.0.98.62:43947/ using document root <none> and password file <none>
I20260620 04:35:53.354113 4983 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:53.354215 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:53.354444 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:53.356513 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "dd0661af56ab4d6a8e70422a33d54798"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "60e99138cc90b3eeee86d393eb4bda43"
server_key_iv: "cd1ce18fed0ddfe4403437d56d693634"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.357074 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd0661af56ab4d6a8e70422a33d54798"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "60e99138cc90b3eeee86d393eb4bda43"
server_key_iv: "cd1ce18fed0ddfe4403437d56d693634"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.360642 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:35:53.363037 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.364446 4983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:53.364584 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "dd0661af56ab4d6a8e70422a33d54798"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "60e99138cc90b3eeee86d393eb4bda43"
server_key_iv: "cd1ce18fed0ddfe4403437d56d693634"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.364698 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:53.369103 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:53.369663 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:53.369846 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:53.376895 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36355
I20260620 04:35:53.376901 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36355 every 8 connection(s)
I20260620 04:35:53.377795 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:53.380838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 4983
I20260620 04:35:53.380981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:53.381023 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:53.381304 392 external_mini_cluster.cc:1428] Setting key 4ac3bb12e6ba99c4c4acf9b9c161f069
I20260620 04:35:53.386937 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.382721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.0.98.62:36355)
0620 04:35:53.383434 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.383452 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.383487 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:35:53.384163 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.384179 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.384593 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.384864 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:53.385700 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:53.385705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:53.385716 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:53.385728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:53.385792 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:53.385808 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:53.385920 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.386035 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.386306 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":136,"threads_started":1}
I20260620 04:35:53.387724 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: Bootstrap starting.
I20260620 04:35:53.390226 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:53.390988 5050 log.cc:824] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:53.393247 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: No bootstrap required, opened a new log
I20260620 04:35:53.397440 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } }
I20260620 04:35:53.397688 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:53.397770 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0661af56ab4d6a8e70422a33d54798, State: Initialized, Role: FOLLOWER
I20260620 04:35:53.398185 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } }
I20260620 04:35:53.398335 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:53.398420 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:53.398540 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:53.399628 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } }
I20260620 04:35:53.399960 5050 leader_election.cc:302] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0661af56ab4d6a8e70422a33d54798; no voters:
I20260620 04:35:53.400336 5050 leader_election.cc:288] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:53.400436 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:53.400691 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [term 1 LEADER]: Becoming Leader. State: Replica: dd0661af56ab4d6a8e70422a33d54798, State: Running, Role: LEADER
I20260620 04:35:53.400985 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } }
I20260620 04:35:53.401516 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:53.402673 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0661af56ab4d6a8e70422a33d54798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } } }
I20260620 04:35:53.402676 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0661af56ab4d6a8e70422a33d54798. Latest consensus state: current_term: 1 leader_uuid: "dd0661af56ab4d6a8e70422a33d54798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0661af56ab4d6a8e70422a33d54798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36355 } } }
I20260620 04:35:53.402908 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:53.402850 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:53.403827 5065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:53.406332 5065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:53.411284 5065 catalog_manager.cc:1347] Generated new cluster ID: 035bca808933456bb2113b508950bb5b
I20260620 04:35:53.411351 5065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:53.440639 5065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:53.440979 5065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:53.449584 5065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: Generated new TSK 0
I20260620 04:35:53.450280 5065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:35:53.458879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:35:53.572899 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:53.573150 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:53.573190 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:53.577478 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:53.577543 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:53.577600 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:53.582440 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:53.583500 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:53.585084 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:53.595530 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.595573 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.595543 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:53.595738 5074 server_base.cc:1034] running on GCE node
I20260620 04:35:53.596459 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:53.597091 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:53.598304 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781930153598274 us; error 25 us; skew 500 ppm
I20260620 04:35:53.600078 5074 webserver.cc:460] Webserver started at http://127.0.98.1:38597/ using document root <none> and password file <none>
I20260620 04:35:53.600703 5074 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:53.600797 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:53.601022 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:53.603097 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "74d6e366685a4f779b923c9c4ff0f0b8"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "85baed0cbcc317cb8724ca65237f33f6"
server_key_iv: "93adb24ea907754088ff8d8838a6b448"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.603647 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74d6e366685a4f779b923c9c4ff0f0b8"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "85baed0cbcc317cb8724ca65237f33f6"
server_key_iv: "93adb24ea907754088ff8d8838a6b448"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.607597 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:35:53.610190 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.611445 5074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:53.611601 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "74d6e366685a4f779b923c9c4ff0f0b8"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "85baed0cbcc317cb8724ca65237f33f6"
server_key_iv: "93adb24ea907754088ff8d8838a6b448"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.611718 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:53.616535 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:53.617141 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:53.617324 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:53.617854 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:53.618726 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:53.618798 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.618866 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:53.618918 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.628389 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36137
I20260620 04:35:53.628445 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36137 every 8 connection(s)
I20260620 04:35:53.629299 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:53.634845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5074
I20260620 04:35:53.634989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:53.635274 392 external_mini_cluster.cc:1428] Setting key af90c72696e93de1ad0ee04f095519dc
I20260620 04:35:53.635339 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.631112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41005 (local address 127.0.98.62:36355)
0620 04:35:53.631263 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.631267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.632302 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 04:35:53.633571 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.633574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.633610 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.633695 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:53.634766 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:53.634769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:53.634772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:53.634784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:53.634811 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:53.634820 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:53.634915 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.635067 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.635110 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:35:53.636588 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.631464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36355 (local address 127.0.98.1:41005)
0620 04:35:53.632107 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:53.632149 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:53.633348 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:53.633844 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:53.633850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:53.633930 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:53.634454 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:53.634456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:53.634493 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:53.634586 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:53.634910 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:53.634918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:53.634925 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:35:53.635096 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:35:53.635409 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260620 04:35:53.637758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 04:35:53.637791 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36355
I20260620 04:35:53.638139 5202 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:53.638521 5202 heartbeater.cc:511] Master 127.0.98.62:36355 requested a full tablet report, sending...
I20260620 04:35:53.640084 5013 ts_manager.cc:194] Registered new tserver with Master: 74d6e366685a4f779b923c9c4ff0f0b8 (127.0.98.1:36137)
W20260620 04:35:53.752934 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:53.753258 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:53.753328 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:53.757719 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:53.757822 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:53.757921 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:53.762821 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:53.763926 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:53.765298 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:53.775391 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.775386 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.775386 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:53.775907 5205 server_base.cc:1034] running on GCE node
I20260620 04:35:53.776361 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:53.776983 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:53.778210 5205 hybrid_clock.cc:648] HybridClock initialized: now 1781930153778177 us; error 45 us; skew 500 ppm
I20260620 04:35:53.780084 5205 webserver.cc:460] Webserver started at http://127.0.98.2:36601/ using document root <none> and password file <none>
I20260620 04:35:53.780679 5205 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:53.780756 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:53.780966 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:53.782997 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "3c1280bf58b9444ea721114bd71aa53d"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "1e5a2b4b531d14dd21929497bf464090"
server_key_iv: "a756c067f508cbc0c32ebb3fa446e5ed"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.783607 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3c1280bf58b9444ea721114bd71aa53d"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "1e5a2b4b531d14dd21929497bf464090"
server_key_iv: "a756c067f508cbc0c32ebb3fa446e5ed"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.787463 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:35:53.790230 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.791568 5205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:35:53.791723 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "3c1280bf58b9444ea721114bd71aa53d"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "1e5a2b4b531d14dd21929497bf464090"
server_key_iv: "a756c067f508cbc0c32ebb3fa446e5ed"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.791846 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:53.796833 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:53.797514 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:53.797712 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:53.798254 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:53.799116 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:53.799201 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.799268 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:53.799316 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.808917 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45807
I20260620 04:35:53.808974 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45807 every 8 connection(s)
I20260620 04:35:53.809811 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:53.814193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5205
I20260620 04:35:53.814386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:53.814703 392 external_mini_cluster.cc:1428] Setting key 3470016179373ef70bb8bebd956c6aba
I20260620 04:35:53.816416 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.811633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36049 (local address 127.0.98.62:36355)
0620 04:35:53.811774 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:53.811777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:53.812812 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 04:35:53.814227 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:53.814230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:53.814274 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:53.814386 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:53.815705 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:53.815709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:53.815712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:53.815724 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:53.815914 (+ 190us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:53.815939 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:53.816045 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:53.816243 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 04:35:53.816288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:35:53.816943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--builtin_ntp_servers=127.0.98.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 04:35:53.817529 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:53.811944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36355 (local address 127.0.98.2:36049)
0620 04:35:53.812668 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:53.812707 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:53.814003 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:53.814612 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:53.814619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:53.814723 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:53.815409 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:53.815411 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:53.815462 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:53.815552 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:53.816057 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:53.816067 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:53.816084 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:35:53.816296 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 04:35:53.816538 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":108,"threads_started":1}
I20260620 04:35:53.819168 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36355
I20260620 04:35:53.819409 5333 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:53.819802 5333 heartbeater.cc:511] Master 127.0.98.62:36355 requested a full tablet report, sending...
I20260620 04:35:53.821005 5013 ts_manager.cc:194] Registered new tserver with Master: 3c1280bf58b9444ea721114bd71aa53d (127.0.98.2:45807)
W20260620 04:35:53.938452 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:53.938799 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:53.938884 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:53.943432 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:53.943539 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:53.943661 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:53.948635 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:53.949944 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:53.951344 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:53.961365 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.961365 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:53.961422 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:53.961758 5336 server_base.cc:1034] running on GCE node
I20260620 04:35:53.962240 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:53.962921 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:53.964128 5336 hybrid_clock.cc:648] HybridClock initialized: now 1781930153964093 us; error 50 us; skew 500 ppm
I20260620 04:35:53.966693 5336 webserver.cc:460] Webserver started at http://127.0.98.3:40495/ using document root <none> and password file <none>
I20260620 04:35:53.967696 5336 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:53.967849 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:53.968137 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:53.970398 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "74895e9b0c6b4e8e902bf96a3f6d078b"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "3da2f59dfdb6ad331deea67d64926ade"
server_key_iv: "9b4944fb4512a601685c3d325fcb4740"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.970970 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74895e9b0c6b4e8e902bf96a3f6d078b"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "3da2f59dfdb6ad331deea67d64926ade"
server_key_iv: "9b4944fb4512a601685c3d325fcb4740"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.975260 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:35:53.978533 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.979861 5336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:35:53.980022 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "74895e9b0c6b4e8e902bf96a3f6d078b"
format_stamp: "Formatted at 2026-06-20 04:35:53 on dist-test-slave-zmf4"
server_key: "3da2f59dfdb6ad331deea67d64926ade"
server_key_iv: "9b4944fb4512a601685c3d325fcb4740"
server_key_version: "encryptionkey@0"
I20260620 04:35:53.980152 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:53.986773 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:53.987735 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:53.987982 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:53.988826 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:53.990203 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:53.990340 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:53.990420 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:53.990468 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:54.000258 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43645
I20260620 04:35:54.000279 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43645 every 8 connection(s)
I20260620 04:35:54.001340 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:54.005035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5336
I20260620 04:35:54.005167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:54.005425 392 external_mini_cluster.cc:1428] Setting key 1788dfb7d79c871937c48c574eb840f4
I20260620 04:35:54.008671 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.003892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36197 (local address 127.0.98.62:36355)
0620 04:35:54.004182 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.004186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.005215 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 04:35:54.006612 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.006615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.006657 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.006721 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.008262 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.008265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.008268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.008281 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.008300 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.008309 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.008389 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.008526 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.008565 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260620 04:35:54.009615 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.004295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36355 (local address 127.0.98.3:36197)
0620 04:35:54.005032 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.005075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.006423 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.006906 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.006914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.007007 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.007905 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.007906 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.007955 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.008028 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.008394 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.008401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.008405 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.008550 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.008763 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260620 04:35:54.010865 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36355
I20260620 04:35:54.011137 5464 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:54.011581 5464 heartbeater.cc:511] Master 127.0.98.62:36355 requested a full tablet report, sending...
I20260620 04:35:54.012815 5013 ts_manager.cc:194] Registered new tserver with Master: 74895e9b0c6b4e8e902bf96a3f6d078b (127.0.98.3:43645)
I20260620 04:35:54.020643 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:35:54.024389 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.022732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.0.98.62:36355)
0620 04:35:54.022880 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.022884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.023050 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:35:54.023202 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.023205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.023241 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.023317 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.023941 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.023943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.023945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.023953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.023966 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.023973 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.024037 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.024119 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.024159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:35:54.027598 5013 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:35:54.031062 5013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:35:54.044291 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.040776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.0.1:47416)
0620 04:35:54.041952 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.041968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.042037 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.042974 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.042979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.042995 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.043488 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.043490 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.043504 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.043535 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.043917 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.043922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.043926 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.044012 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.044069 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":124,"threads_started":1}
I20260620 04:35:54.044550 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.040666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43645 (local address 127.0.0.1:42846)
0620 04:35:54.041498 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.041570 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.041670 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.042424 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.042431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.042545 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.043332 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.043334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.043357 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.043440 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.043817 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.043827 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.043833 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.043953 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.044040 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":94,"threads_started":1}
I20260620 04:35:54.044792 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.040782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.0.98.3:43645)
0620 04:35:54.041927 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.041944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.041972 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:35:54.042032 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.042037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.042253 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.042398 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.043583 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.043590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.043604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.043618 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.043667 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.043698 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.044468 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.044524 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.044608 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":128,"threads_started":1}
I20260620 04:35:54.045084 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.041478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.98.2:45807)
0620 04:35:54.042352 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.042365 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.042388 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:35:54.042441 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.042446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.042605 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.042753 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.043657 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.043662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.043670 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.043682 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.043721 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.043752 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.043863 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.044835 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.044909 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":167,"threads_started":1}
I20260620 04:35:54.046473 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.041640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36137 (local address 127.0.0.1:42472)
0620 04:35:54.042081 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.042099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.042220 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.044154 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.044157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.044171 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.045753 (+ 1582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.045755 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.045769 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.045803 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.046217 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.046220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.046224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.046316 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.046354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":103,"threads_started":1}
I20260620 04:35:54.046813 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.042177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.98.1:36137)
0620 04:35:54.042673 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.042689 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.042719 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:35:54.042787 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.042794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.043672 (+ 878us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.043862 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.045989 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.045997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.046010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.046025 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.046071 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.046105 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.046229 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.046340 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.046418 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":143,"threads_started":1}
I20260620 04:35:54.048144 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 80320457372b4d749f9fb748897e72f5 (DEFAULT_TABLE table=test-table [id=8858b866851844d894ea7e2d34a2745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:54.048143 5397 tablet_service.cc:1467] Processing CreateTablet for tablet 80320457372b4d749f9fb748897e72f5 (DEFAULT_TABLE table=test-table [id=8858b866851844d894ea7e2d34a2745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:54.049214 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80320457372b4d749f9fb748897e72f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:54.049226 5397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80320457372b4d749f9fb748897e72f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:54.049980 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 80320457372b4d749f9fb748897e72f5 (DEFAULT_TABLE table=test-table [id=8858b866851844d894ea7e2d34a2745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:35:54.051002 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80320457372b4d749f9fb748897e72f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:54.055361 5481 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Bootstrap starting.
I20260620 04:35:54.057518 5482 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Bootstrap starting.
I20260620 04:35:54.058672 5481 tablet_bootstrap.cc:654] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:54.059530 5483 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Bootstrap starting.
I20260620 04:35:54.059962 5481 log.cc:824] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:54.060334 5482 tablet_bootstrap.cc:654] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:54.061233 5482 log.cc:824] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:54.062206 5483 tablet_bootstrap.cc:654] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:54.062296 5481 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: No bootstrap required, opened a new log
I20260620 04:35:54.062516 5481 ts_tablet_manager.cc:1397] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:35:54.063045 5483 log.cc:824] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:54.065120 5483 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: No bootstrap required, opened a new log
I20260620 04:35:54.065120 5482 tablet_bootstrap.cc:492] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: No bootstrap required, opened a new log
I20260620 04:35:54.065325 5483 ts_tablet_manager.cc:1397] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:35:54.065325 5482 ts_tablet_manager.cc:1397] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260620 04:35:54.066555 5481 raft_consensus.cc:348] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.066882 5481 raft_consensus.cc:374] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:54.066936 5481 raft_consensus.cc:729] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1280bf58b9444ea721114bd71aa53d, State: Initialized, Role: FOLLOWER
I20260620 04:35:54.067446 5481 consensus_queue.cc:260] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.068476 5333 heartbeater.cc:503] Master 127.0.98.62:36355 was elected leader, sending a full tablet report...
I20260620 04:35:54.068609 5481 ts_tablet_manager.cc:1428] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:35:54.070205 5482 raft_consensus.cc:348] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.070205 5483 raft_consensus.cc:348] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.070483 5482 raft_consensus.cc:374] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:54.070483 5483 raft_consensus.cc:374] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:54.070544 5483 raft_consensus.cc:729] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d6e366685a4f779b923c9c4ff0f0b8, State: Initialized, Role: FOLLOWER
I20260620 04:35:54.070552 5482 raft_consensus.cc:729] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74895e9b0c6b4e8e902bf96a3f6d078b, State: Initialized, Role: FOLLOWER
I20260620 04:35:54.071013 5483 consensus_queue.cc:260] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.071013 5482 consensus_queue.cc:260] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.071858 5202 heartbeater.cc:503] Master 127.0.98.62:36355 was elected leader, sending a full tablet report...
I20260620 04:35:54.072062 5483 ts_tablet_manager.cc:1428] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 04:35:54.072062 5482 ts_tablet_manager.cc:1428] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 04:35:54.072417 5464 heartbeater.cc:503] Master 127.0.98.62:36355 was elected leader, sending a full tablet report...
W20260620 04:35:54.130668 5203 tablet.cc:2369] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:35:54.252713 5465 tablet.cc:2369] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:35:54.311012 5334 tablet.cc:2369] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:35:54.321174 5488 raft_consensus.cc:482] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:35:54.321393 5488 raft_consensus.cc:504] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.322494 5488 leader_election.cc:288] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c1280bf58b9444ea721114bd71aa53d (127.0.98.2:45807), 74d6e366685a4f779b923c9c4ff0f0b8 (127.0.98.1:36137)
I20260620 04:35:54.325209 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.323037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45807 (local address 127.0.98.3:37305)
0620 04:35:54.323260 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.323280 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.323437 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.323760 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.323764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.323794 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.324485 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.324486 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.324497 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.324524 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.324935 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.324939 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.324943 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.325046 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.325092 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 04:35:54.325294 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.323275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37305 (local address 127.0.98.2:45807)
0620 04:35:54.323446 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.323452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.323473 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:35:54.323550 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.323553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.323602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.323696 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.324704 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.324711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.324716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.324735 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.324769 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.324783 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.324900 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.325069 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.325136 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 04:35:54.325541 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.323387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36137 (local address 127.0.98.3:38911)
0620 04:35:54.323750 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.323784 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.323879 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.324160 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.324164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.324182 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:54.324956 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:35:54.324958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:54.324970 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:35:54.325002 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:54.325273 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:54.325276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:54.325280 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:35:54.325362 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:35:54.325400 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260620 04:35:54.325806 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.323585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38911 (local address 127.0.98.1:36137)
0620 04:35:54.323732 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.323737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.323888 (+ 151us) server_negotiation.cc:407] Connection header received
0620 04:35:54.323955 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.323961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.324009 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.324102 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.325127 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.325131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.325133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.325145 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.325165 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.325174 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.325267 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.325655 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.325698 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:35:54.325868 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80320457372b4d749f9fb748897e72f5" candidate_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1280bf58b9444ea721114bd71aa53d" is_pre_election: true
I20260620 04:35:54.326185 5287 raft_consensus.cc:2454] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74895e9b0c6b4e8e902bf96a3f6d078b in term 0.
I20260620 04:35:54.326274 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80320457372b4d749f9fb748897e72f5" candidate_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" is_pre_election: true
I20260620 04:35:54.326577 5156 raft_consensus.cc:2454] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74895e9b0c6b4e8e902bf96a3f6d078b in term 0.
I20260620 04:35:54.326659 5354 leader_election.cc:302] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c1280bf58b9444ea721114bd71aa53d, 74895e9b0c6b4e8e902bf96a3f6d078b; no voters:
I20260620 04:35:54.326941 5488 raft_consensus.cc:2781] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:35:54.327025 5488 raft_consensus.cc:482] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:35:54.327068 5488 raft_consensus.cc:3037] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:54.328109 5488 raft_consensus.cc:504] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.328513 5488 leader_election.cc:288] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [CANDIDATE]: Term 1 election: Requested vote from peers 3c1280bf58b9444ea721114bd71aa53d (127.0.98.2:45807), 74d6e366685a4f779b923c9c4ff0f0b8 (127.0.98.1:36137)
I20260620 04:35:54.328888 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80320457372b4d749f9fb748897e72f5" candidate_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d6e366685a4f779b923c9c4ff0f0b8"
I20260620 04:35:54.328888 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80320457372b4d749f9fb748897e72f5" candidate_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c1280bf58b9444ea721114bd71aa53d"
I20260620 04:35:54.328995 5156 raft_consensus.cc:3037] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:54.329023 5287 raft_consensus.cc:3037] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:54.330166 5156 raft_consensus.cc:2454] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74895e9b0c6b4e8e902bf96a3f6d078b in term 1.
I20260620 04:35:54.330175 5287 raft_consensus.cc:2454] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74895e9b0c6b4e8e902bf96a3f6d078b in term 1.
I20260620 04:35:54.330441 5354 leader_election.cc:302] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74895e9b0c6b4e8e902bf96a3f6d078b, 74d6e366685a4f779b923c9c4ff0f0b8; no voters:
I20260620 04:35:54.330617 5488 raft_consensus.cc:2781] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:54.330826 5488 raft_consensus.cc:686] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 1 LEADER]: Becoming Leader. State: Replica: 74895e9b0c6b4e8e902bf96a3f6d078b, State: Running, Role: LEADER
I20260620 04:35:54.331192 5488 consensus_queue.cc:237] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } }
I20260620 04:35:54.334331 5013 catalog_manager.cc:5495] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b reported cstate change: term changed from 0 to 1, leader changed from <none> to 74895e9b0c6b4e8e902bf96a3f6d078b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74895e9b0c6b4e8e902bf96a3f6d078b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 } health_report { overall_health: UNKNOWN } } }
I20260620 04:35:54.345960 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.344462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.98.3:43645)
0620 04:35:54.344597 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.344600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.344614 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:35:54.344750 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.344752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.344810 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.344882 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.345617 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.345620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.345622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.345632 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.345649 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.345657 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.345732 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.345802 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.345833 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:35:54.351923 5156 raft_consensus.cc:1261] T 80320457372b4d749f9fb748897e72f5 P 74d6e366685a4f779b923c9c4ff0f0b8 [term 1 FOLLOWER]: Refusing update from remote peer 74895e9b0c6b4e8e902bf96a3f6d078b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:54.351979 5287 raft_consensus.cc:1261] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 1 FOLLOWER]: Refusing update from remote peer 74895e9b0c6b4e8e902bf96a3f6d078b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:35:54.352710 5488 consensus_queue.cc:1035] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1280bf58b9444ea721114bd71aa53d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:54.352909 5491 consensus_queue.cc:1035] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d6e366685a4f779b923c9c4ff0f0b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:35:54.360785 5495 mvcc.cc:205] Tried to move back new op lower bound from 7298785912218324992 to 7298785912143179776. Current Snapshot: MvccSnapshot[applied={T|T < 7298785912218324992}]
I20260620 04:35:54.369675 5012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:54.369863 5012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:35:54.372444 5012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd0661af56ab4d6a8e70422a33d54798: Sending DeleteTablet for 3 replicas of tablet 80320457372b4d749f9fb748897e72f5
I20260620 04:35:54.373217 5397 tablet_service.cc:1514] Processing DeleteTablet for tablet 80320457372b4d749f9fb748897e72f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:54 UTC) from {username='slave'} at 127.0.0.1:42846
I20260620 04:35:54.373322 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 80320457372b4d749f9fb748897e72f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:54 UTC) from {username='slave'} at 127.0.0.1:42472
I20260620 04:35:54.373409 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet 80320457372b4d749f9fb748897e72f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:35:54 UTC) from {username='slave'} at 127.0.0.1:47416
I20260620 04:35:54.373881 5506 tablet_replica.cc:330] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: stopping tablet replica
I20260620 04:35:54.374153 5504 tablet_replica.cc:330] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: stopping tablet replica
I20260620 04:35:54.374207 5506 raft_consensus.cc:2229] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:35:54.374235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5074
I20260620 04:35:54.375476 5506 raft_consensus.cc:2258] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:54.377110 5506 ts_tablet_manager.cc:1905] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:54.380591 5506 ts_tablet_manager.cc:1918] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:54.380730 5506 log.cc:1196] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/80320457372b4d749f9fb748897e72f5
I20260620 04:35:54.381083 5506 ts_tablet_manager.cc:1939] T 80320457372b4d749f9fb748897e72f5 P 3c1280bf58b9444ea721114bd71aa53d: Deleting consensus metadata
W20260620 04:35:54.382283 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36137 (error 108)
I20260620 04:35:54.382714 5001 catalog_manager.cc:4841] TS 3c1280bf58b9444ea721114bd71aa53d (127.0.98.2:45807): tablet 80320457372b4d749f9fb748897e72f5 (table test-table [id=8858b866851844d894ea7e2d34a2745c]) successfully deleted
I20260620 04:35:54.383239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5205
I20260620 04:35:54.388481 5504 raft_consensus.cc:2229] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:35:54.388967 5504 raft_consensus.cc:2258] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:35:54.390040 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.389709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36137 (local address 127.0.0.1:42478)
0620 04:35:54.389889 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.389959 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 04:35:54.390288 5001 catalog_manager.cc:4584] TS 74d6e366685a4f779b923c9c4ff0f0b8 (127.0.98.1:36137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80320457372b4d749f9fb748897e72f5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36137: connect: Connection refused (error 111)
I20260620 04:35:54.390658 5504 ts_tablet_manager.cc:1905] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:35:54.393150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5336
I20260620 04:35:54.393216 5504 ts_tablet_manager.cc:1918] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:35:54.393301 5504 log.cc:1196] T 80320457372b4d749f9fb748897e72f5 P 74895e9b0c6b4e8e902bf96a3f6d078b: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/80320457372b4d749f9fb748897e72f5
I20260620 04:35:54.398790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 4983
2026-06-20T04:35:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1213 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 04:35:54.420717 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:35:54.422289 392 webserver.cc:460] Webserver started at https://127.0.0.1:38285/ using document root <none> and password file <none>
I20260620 04:35:54.422473 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:35:54.422752 392 webserver.cc:460] Webserver started at http://127.0.0.1:36919/ using document root <none> and password file <none>
2026-06-20T04:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:54Z Disabled control of system clock
I20260620 04:35:54.431967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/testchainca.cert with env {}
W20260620 04:35:54.542263 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:54.542500 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:54.542537 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:54.546614 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:54.546661 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:54.546676 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:54.546693 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:54.546708 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:35:54.546743 5513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:35:54.546773 5513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38285/jwks.json
I20260620 04:35:54.551883 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:54.552868 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:54.553795 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:54.561939 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:54.561973 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:54.561983 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:54.562120 5513 server_base.cc:1034] running on GCE node
I20260620 04:35:54.562788 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:54.563812 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:54.565013 5513 hybrid_clock.cc:648] HybridClock initialized: now 1781930154564999 us; error 26 us; skew 500 ppm
I20260620 04:35:54.566906 5513 webserver.cc:460] Webserver started at http://127.0.98.62:38441/ using document root <none> and password file <none>
I20260620 04:35:54.567467 5513 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:54.567575 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:54.567777 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:54.569736 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "7c12e2fbbb984f01809f7b142a3261c6"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "47833a778a4445bb2b3ce3496e6e595e"
server_key_iv: "2c3ef14199b4174287349491321f2fa1"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.570266 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c12e2fbbb984f01809f7b142a3261c6"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "47833a778a4445bb2b3ce3496e6e595e"
server_key_iv: "2c3ef14199b4174287349491321f2fa1"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.573793 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 04:35:54.576117 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:54.577448 5513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:35:54.577589 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "7c12e2fbbb984f01809f7b142a3261c6"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "47833a778a4445bb2b3ce3496e6e595e"
server_key_iv: "2c3ef14199b4174287349491321f2fa1"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.577697 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:54.616618 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:54.617360 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:54.617551 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:54.624953 5513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45991
I20260620 04:35:54.624981 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45991 every 8 connection(s)
I20260620 04:35:54.625874 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:54.627754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5513
I20260620 04:35:54.627857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:54.628171 392 external_mini_cluster.cc:1428] Setting key 6da9105da06e6f910116c96344447374
I20260620 04:35:54.629243 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:54.635098 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6: Bootstrap starting.
I20260620 04:35:54.637480 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:54.637598 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.629569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.0.98.62:45991)
0620 04:35:54.630214 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.630230 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.630301 (+ 71us) server_negotiation.cc:407] Connection header received
0620 04:35:54.631038 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.631054 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.631453 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.631959 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.632485 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.633410 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.634286 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.634743 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.634908 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:54.635851 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.635856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.635867 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.636287 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.636340 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.636360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.636463 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.636605 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.636845 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260620 04:35:54.638358 5580 log.cc:824] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:54.640398 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6: No bootstrap required, opened a new log
I20260620 04:35:54.640646 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:35:54.643821 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260620 04:35:54.644060 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:54.644124 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c12e2fbbb984f01809f7b142a3261c6, State: Initialized, Role: FOLLOWER
I20260620 04:35:54.644532 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260620 04:35:54.644678 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:54.644768 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:54.644863 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:54.645674 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260620 04:35:54.645993 5580 leader_election.cc:302] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c12e2fbbb984f01809f7b142a3261c6; no voters:
I20260620 04:35:54.646275 5580 leader_election.cc:288] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:54.646428 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:54.646648 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [term 1 LEADER]: Becoming Leader. State: Replica: 7c12e2fbbb984f01809f7b142a3261c6, State: Running, Role: LEADER
I20260620 04:35:54.646950 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260620 04:35:54.647429 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:54.648059 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c12e2fbbb984f01809f7b142a3261c6. Latest consensus state: current_term: 1 leader_uuid: "7c12e2fbbb984f01809f7b142a3261c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260620 04:35:54.648222 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:54.648453 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c12e2fbbb984f01809f7b142a3261c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c12e2fbbb984f01809f7b142a3261c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260620 04:35:54.648561 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:54.648957 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:54.651939 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:54.657377 5594 catalog_manager.cc:1347] Generated new cluster ID: 8a649f669dd0465cb775ff03c869ec2b
I20260620 04:35:54.657476 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:54.675150 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:54.676015 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:54.683072 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c12e2fbbb984f01809f7b142a3261c6: Generated new TSK 0
I20260620 04:35:54.683769 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:35:54.717159 5610 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 04:35:54.717284 5610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:35:54.717805 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.700740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.98.62:45991)
0620 04:35:54.700867 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.700872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.701037 (+ 165us) server_negotiation.cc:407] Connection header received
0620 04:35:54.701219 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.701222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.701274 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.701377 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:35:54.701593 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.702380 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.703265 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.703599 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.703694 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:54.704030 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:35:54.717409 (+ 13379us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:35:54.717662 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 04:35:54.717866 5583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 04:35:54.718884 5609 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:54.700625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45991 (local address 127.0.0.1:45860)
0620 04:35:54.700959 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.700972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.701081 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.701391 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.701394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.701400 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:35:54.701499 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:54.701504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.702528 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.702531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:54.703152 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:54.703159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.703711 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.703726 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:54.703815 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:54.703856 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:35:54.717643 (+ 13787us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:35:54.717827 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 04:35:54.719564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5513
2026-06-20T04:35:54Z chronyd exiting
I20260620 04:35:54.756685 392 test_util.cc:175] -----------------------------------------------
I20260620 04:35:54.756763 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781930128116734-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (337 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 04:35:54.757759 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:35:54.758602 392 webserver.cc:460] Webserver started at https://127.0.0.1:37017/ using document root <none> and password file <none>
I20260620 04:35:54.758682 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:35:54.758957 392 webserver.cc:460] Webserver started at http://127.0.0.1:45999/ using document root <none> and password file <none>
2026-06-20T04:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:54Z Disabled control of system clock
I20260620 04:35:54.767314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34313
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34313
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/testchainca.cert with env {}
W20260620 04:35:54.872952 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:54.873241 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:54.873314 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:54.877450 5616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:54.877547 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:54.877584 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:54.877650 5616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:54.877694 5616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:35:54.877732 5616 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:35:54.877759 5616 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37017/jwks.json
I20260620 04:35:54.882998 5616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:54.884025 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:54.885033 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:54.893512 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:54.893513 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:54.893656 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:54.893919 5616 server_base.cc:1034] running on GCE node
I20260620 04:35:54.894449 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:54.895498 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:54.896701 5616 hybrid_clock.cc:648] HybridClock initialized: now 1781930154896650 us; error 49 us; skew 500 ppm
I20260620 04:35:54.898532 5616 webserver.cc:460] Webserver started at http://127.0.98.62:36007/ using document root <none> and password file <none>
I20260620 04:35:54.899085 5616 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:54.899184 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:54.899403 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:54.901413 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "dd8ea3c5221b44e599aadd4ae112dc26"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "5eb838a909274ffbafbece2ad9eec849"
server_key_iv: "9d87bbc5c81d9821ad145f5bf2236273"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.901978 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd8ea3c5221b44e599aadd4ae112dc26"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "5eb838a909274ffbafbece2ad9eec849"
server_key_iv: "9d87bbc5c81d9821ad145f5bf2236273"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.905776 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 04:35:54.908345 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:54.909682 5616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:54.909828 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "dd8ea3c5221b44e599aadd4ae112dc26"
format_stamp: "Formatted at 2026-06-20 04:35:54 on dist-test-slave-zmf4"
server_key: "5eb838a909274ffbafbece2ad9eec849"
server_key_iv: "9d87bbc5c81d9821ad145f5bf2236273"
server_key_version: "encryptionkey@0"
I20260620 04:35:54.909945 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:54.924688 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:54.925345 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:54.925544 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:54.932484 5616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34313
I20260620 04:35:54.932546 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34313 every 8 connection(s)
I20260620 04:35:54.933429 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:54.936547 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:54.941813 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26: Bootstrap starting.
I20260620 04:35:54.942703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5616
I20260620 04:35:54.942821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:54.943109 392 external_mini_cluster.cc:1428] Setting key 74921283230d65d18594e400f3c4e263
I20260620 04:35:54.944638 5683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:54.945356 5683 log.cc:824] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:54.947149 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26: No bootstrap required, opened a new log
I20260620 04:35:54.950563 5683 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } }
I20260620 04:35:54.950762 5683 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:54.950824 5683 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8ea3c5221b44e599aadd4ae112dc26, State: Initialized, Role: FOLLOWER
I20260620 04:35:54.951256 5683 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } }
I20260620 04:35:54.951370 5683 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:54.951449 5683 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:54.951545 5683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:54.952499 5683 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } }
I20260620 04:35:54.952759 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.944239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.98.62:34313)
0620 04:35:54.944971 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.944986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.945022 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:35:54.945800 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.945818 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.946144 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.946497 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:54.946927 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.947999 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.949118 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:54.949585 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:54.949758 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:54.950721 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:54.950726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:54.950737 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:54.951252 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:54.951303 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:54.951322 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:54.951438 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:54.951655 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 04:35:54.951946 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260620 04:35:54.952821 5683 leader_election.cc:302] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8ea3c5221b44e599aadd4ae112dc26; no voters:
I20260620 04:35:54.953161 5683 leader_election.cc:288] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:54.953233 5688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:54.953397 5688 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [term 1 LEADER]: Becoming Leader. State: Replica: dd8ea3c5221b44e599aadd4ae112dc26, State: Running, Role: LEADER
I20260620 04:35:54.953671 5688 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } }
I20260620 04:35:54.954802 5683 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:54.955660 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } } }
I20260620 04:35:54.955796 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:54.955832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:35:54.956091 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd8ea3c5221b44e599aadd4ae112dc26. Latest consensus state: current_term: 1 leader_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8ea3c5221b44e599aadd4ae112dc26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34313 } } }
I20260620 04:35:54.956173 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:54.956439 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:54.959343 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:54.964972 5697 catalog_manager.cc:1347] Generated new cluster ID: 6bb9d424172449f8b5f0560c6e3d0f6e
I20260620 04:35:54.965045 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:54.971882 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:54.973073 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:54.985529 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd8ea3c5221b44e599aadd4ae112dc26: Generated new TSK 0
I20260620 04:35:54.986295 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:35:55.014860 5713 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 04:35:55.014948 5713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:35:55.015571 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:54.999339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.62:34313)
0620 04:35:54.999474 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:54.999478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:54.999652 (+ 174us) server_negotiation.cc:407] Connection header received
0620 04:35:54.999776 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:54.999780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:54.999836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:54.999954 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:35:55.000241 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.001081 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.001958 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.002331 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.002442 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.002758 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:35:55.015158 (+ 12400us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:35:55.015420 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:35:55.015647 5686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 04:35:55.016726 5712 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:54.999237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34313 (local address 127.0.0.1:38872)
0620 04:35:54.999563 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:54.999575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:54.999670 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:54.999984 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:54.999986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:54.999998 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:35:55.000095 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.000100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.001215 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.001217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.001834 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.001841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.002463 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.002479 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.002552 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.002579 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:35:55.016421 (+ 13842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:35:55.016592 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34313: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260620 04:35:55.017577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5616
2026-06-20T04:35:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 04:35:55.056985 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:35:55.057816 392 webserver.cc:460] Webserver started at https://127.0.0.1:41045/ using document root <none> and password file <none>
I20260620 04:35:55.057893 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:35:55.058094 392 webserver.cc:460] Webserver started at http://127.0.0.1:45365/ using document root <none> and password file <none>
2026-06-20T04:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:55Z Disabled control of system clock
I20260620 04:35:55.067180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39453
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41045/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:35:55.173640 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:55.173879 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:55.173916 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:55.177732 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:55.177781 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:55.177798 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:55.177814 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:55.177829 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:35:55.177843 5719 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:35:55.177857 5719 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41045/jwks.json
I20260620 04:35:55.182334 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41045/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:55.183240 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:55.184463 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:55.192394 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:55.192482 5719 server_base.cc:1034] running on GCE node
W20260620 04:35:55.192452 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:55.192405 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:55.193125 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:55.194144 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:55.195341 5719 hybrid_clock.cc:648] HybridClock initialized: now 1781930155195319 us; error 36 us; skew 500 ppm
I20260620 04:35:55.197849 5719 webserver.cc:460] Webserver started at http://127.0.98.62:39457/ using document root <none> and password file <none>
I20260620 04:35:55.198436 5719 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:55.198498 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:55.198707 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:55.200736 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "1053802b503ca8538b1c65e4cdc1ea27"
server_key_iv: "21ea9b3e3383f98caf1d317227a1e503"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.201289 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "1053802b503ca8538b1c65e4cdc1ea27"
server_key_iv: "21ea9b3e3383f98caf1d317227a1e503"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.204821 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:55.207280 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:55.208648 5719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:55.208786 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "1053802b503ca8538b1c65e4cdc1ea27"
server_key_iv: "21ea9b3e3383f98caf1d317227a1e503"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.208894 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:55.223559 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:55.224162 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:55.224391 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:55.231071 5719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39453
I20260620 04:35:55.231101 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39453 every 8 connection(s)
I20260620 04:35:55.231993 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:55.232705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5719
I20260620 04:35:55.232837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:55.233093 392 external_mini_cluster.cc:1428] Setting key 3a79aa017a168279a1364fcee7ebc00d
I20260620 04:35:55.236080 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:55.242574 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964: Bootstrap starting.
I20260620 04:35:55.243546 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:55.234347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.98.62:39453)
0620 04:35:55.235025 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:55.235044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:55.235087 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:35:55.236003 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:55.236027 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:55.236575 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:55.236944 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:55.237472 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.238896 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.239888 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.240571 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.240744 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.241602 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:55.241609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:55.241625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:55.242085 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:55.242150 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:55.242175 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:55.242272 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:55.242452 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 04:35:55.242715 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260620 04:35:55.244892 5786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:55.245690 5786 log.cc:824] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:55.246325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:35:55.247452 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964: No bootstrap required, opened a new log
I20260620 04:35:55.250941 5786 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260620 04:35:55.251201 5786 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:55.251395 5786 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1dc6c8b2d1c4857a0c49e9bab4f5964, State: Initialized, Role: FOLLOWER
I20260620 04:35:55.251927 5786 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260620 04:35:55.252056 5786 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:55.252099 5786 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:55.252166 5786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:55.253053 5786 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260620 04:35:55.253346 5786 leader_election.cc:302] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1dc6c8b2d1c4857a0c49e9bab4f5964; no voters:
I20260620 04:35:55.253597 5786 leader_election.cc:288] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:55.253756 5791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:55.253996 5791 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [term 1 LEADER]: Becoming Leader. State: Replica: a1dc6c8b2d1c4857a0c49e9bab4f5964, State: Running, Role: LEADER
I20260620 04:35:55.254285 5791 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260620 04:35:55.254881 5786 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:55.256325 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } } }
I20260620 04:35:55.256458 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:55.256336 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1dc6c8b2d1c4857a0c49e9bab4f5964. Latest consensus state: current_term: 1 leader_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dc6c8b2d1c4857a0c49e9bab4f5964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } } }
I20260620 04:35:55.256786 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:55.257182 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:55.260324 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:55.266165 5800 catalog_manager.cc:1347] Generated new cluster ID: 31f1f1019e894b6aa2e8cba33fd5c0a1
I20260620 04:35:55.266258 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:55.277742 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:55.278636 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:55.286682 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1dc6c8b2d1c4857a0c49e9bab4f5964: Generated new TSK 0
I20260620 04:35:55.287369 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:35:55.303568 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:55.298169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.0.98.62:39453)
0620 04:35:55.298296 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:55.298305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:55.298585 (+ 280us) server_negotiation.cc:407] Connection header received
0620 04:35:55.298734 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:55.298737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:55.298811 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:55.298952 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:35:55.299294 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.299856 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.301141 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.301362 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.301446 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.301806 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:35:55.303210 (+ 1404us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:35:55.303427 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:35:55.303635 5789 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 04:35:55.305680 5815 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:55.298100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39453 (local address 127.0.0.1:52802)
0620 04:35:55.298435 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:55.298451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:55.298566 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:55.298982 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:55.298985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:55.298993 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:35:55.299116 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.299123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.299977 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.299981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.300991 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.301011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.301472 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.301493 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.301582 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.301610 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:35:55.303380 (+ 1770us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:35:55.305506 (+ 2126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41045/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1860,"thread_start_us":100,"threads_started":1}
I20260620 04:35:55.306262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5719
E20260620 04:35:55.315263 5816 webserver.cc:533] Webserver: sslize failed:
E20260620 04:35:55.315332 5816 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T04:35:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T04:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:55Z Disabled control of system clock
I20260620 04:35:55.353866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37231
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37231
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/testchainca.cert with env {}
W20260620 04:35:55.464448 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:55.464699 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:55.464736 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:55.468892 5820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:55.468950 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:55.468966 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:55.468981 5820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:55.468997 5820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:35:55.469233 5820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:35:55.469311 5820 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 04:35:55.474783 5820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:55.475836 5820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:55.477092 5820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:55.485586 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:55.485611 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:55.485656 5820 server_base.cc:1034] running on GCE node
W20260620 04:35:55.485586 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:55.486320 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:55.487295 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:55.488504 5820 hybrid_clock.cc:648] HybridClock initialized: now 1781930155488457 us; error 60 us; skew 500 ppm
I20260620 04:35:55.490445 5820 webserver.cc:460] Webserver started at http://127.0.98.62:42411/ using document root <none> and password file <none>
I20260620 04:35:55.491004 5820 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:55.491065 5820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:55.491266 5820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:55.493260 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "01b49926f2bb43bc89b1f0553cdacd9b"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "45a55bba8e35da2694d171260a971a54"
server_key_iv: "87b790ffc1ac3af985d20e46a375888e"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.493824 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "01b49926f2bb43bc89b1f0553cdacd9b"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "45a55bba8e35da2694d171260a971a54"
server_key_iv: "87b790ffc1ac3af985d20e46a375888e"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.497377 5820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:55.499979 5834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:55.501247 5820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:55.501391 5820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "01b49926f2bb43bc89b1f0553cdacd9b"
format_stamp: "Formatted at 2026-06-20 04:35:55 on dist-test-slave-zmf4"
server_key: "45a55bba8e35da2694d171260a971a54"
server_key_iv: "87b790ffc1ac3af985d20e46a375888e"
server_key_version: "encryptionkey@0"
I20260620 04:35:55.501504 5820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:55.531124 5820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:55.531812 5820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:55.532008 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:55.538950 5820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37231
I20260620 04:35:55.538962 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37231 every 8 connection(s)
I20260620 04:35:55.540207 5820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:55.543151 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:55.548161 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: Bootstrap starting.
I20260620 04:35:55.550237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5820
I20260620 04:35:55.550375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:55.550575 5887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:55.550684 392 external_mini_cluster.cc:1428] Setting key 6f8f7190a41ff00cbefb5b0c20bd307e
I20260620 04:35:55.551383 5887 log.cc:824] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:55.553723 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: No bootstrap required, opened a new log
I20260620 04:35:55.557971 5887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } }
I20260620 04:35:55.558190 5887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:55.558252 5887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b49926f2bb43bc89b1f0553cdacd9b, State: Initialized, Role: FOLLOWER
I20260620 04:35:55.558755 5887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } }
I20260620 04:35:55.558910 5887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:55.558976 5887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:55.559080 5887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:55.560142 5887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } }
I20260620 04:35:55.560491 5887 leader_election.cc:302] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b49926f2bb43bc89b1f0553cdacd9b; no voters:
I20260620 04:35:55.560763 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:55.551685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.0.98.62:37231)
0620 04:35:55.552361 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:55.552380 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:55.552422 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:35:55.553237 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:55.553257 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:55.553638 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:55.553964 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:55.554385 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.555890 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.556911 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.557657 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.557816 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.558682 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:55.558688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:55.558702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:35:55.559190 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:55.559256 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:35:55.559285 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:55.559383 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:55.559574 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 04:35:55.559826 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260620 04:35:55.560806 5887 leader_election.cc:288] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:55.560961 5892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:55.561287 5892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [term 1 LEADER]: Becoming Leader. State: Replica: 01b49926f2bb43bc89b1f0553cdacd9b, State: Running, Role: LEADER
I20260620 04:35:55.561600 5892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } }
I20260620 04:35:55.561949 5887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:55.562913 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b49926f2bb43bc89b1f0553cdacd9b. Latest consensus state: current_term: 1 leader_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } } }
I20260620 04:35:55.563118 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:55.563390 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b49926f2bb43bc89b1f0553cdacd9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37231 } } }
I20260620 04:35:55.563503 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:55.563771 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 04:35:55.566692 5907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:35:55.566762 5907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:35:55.566850 5901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:55.567566 5901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:55.573145 5901 catalog_manager.cc:1347] Generated new cluster ID: 556c7077bc42428f848e895623574d39
I20260620 04:35:55.573210 5901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:55.588428 5901 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:55.589818 5901 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:55.599828 5901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b49926f2bb43bc89b1f0553cdacd9b: Generated new TSK 0
I20260620 04:35:55.600642 5901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:35:55.651592 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:55.614193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.98.62:37231)
0620 04:35:55.614331 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:55.614336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:55.614471 (+ 135us) server_negotiation.cc:407] Connection header received
0620 04:35:55.614613 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:55.614617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:55.614672 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:55.614780 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:35:55.615053 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.615862 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.617323 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.617673 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.617781 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.618093 (+ 312us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:35:55.650704 (+ 32611us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:35:55.651438 (+ 734us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45480: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 04:35:55.651623 5916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:55.614077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37231 (local address 127.0.0.1:45480)
0620 04:35:55.614382 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:55.614394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:55.614483 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:55.614814 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:55.614817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:55.614824 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:35:55.614923 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.614928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.616010 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.616075 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.617205 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:55.617215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:55.617798 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:55.617823 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:55.617890 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:55.617912 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:35:55.651295 (+ 33383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:35:55.651473 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37231: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20260620 04:35:55.651679 5891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45480: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 04:35:55.652345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5820
2026-06-20T04:35:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (324 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:55 dist-test-slave-zmf4 krb5kdc[5928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:55 dist-test-slave-zmf4 krb5kdc[5928](info): set up 2 sockets
Jun 20 04:35:55 dist-test-slave-zmf4 krb5kdc[5928](info): commencing operation
krb5kdc: starting...
W20260620 04:35:56.964884 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:35:57.139424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:57.251881 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:57.252177 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:57.252270 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:57.256289 5941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:57.256376 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:57.256414 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:57.256470 5941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:57.256510 5941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:57.262048 5941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:57.263245 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:57.264321 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:57.273450 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.273613 5941 server_base.cc:1034] running on GCE node
W20260620 04:35:57.273470 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.273464 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.274358 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:57.275523 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:57.276710 5941 hybrid_clock.cc:648] HybridClock initialized: now 1781930157276680 us; error 41 us; skew 500 ppm
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:57.287624 5941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:35:57.288596 5941 webserver.cc:460] Webserver started at http://127.0.98.62:33821/ using document root <none> and password file <none>
I20260620 04:35:57.289153 5941 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:57.289215 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:57.289433 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:57.291445 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "79a4a5f7fb74478d96d95a26a60517ff"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "a4de6d2c16305a31669483a8ee6e9ce8"
server_key_iv: "d8cf4bf058220098b5ce6616bcf06a8e"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.292001 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "79a4a5f7fb74478d96d95a26a60517ff"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "a4de6d2c16305a31669483a8ee6e9ce8"
server_key_iv: "d8cf4bf058220098b5ce6616bcf06a8e"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.295895 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:35:57.298377 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.299660 5941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:35:57.299862 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "79a4a5f7fb74478d96d95a26a60517ff"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "a4de6d2c16305a31669483a8ee6e9ce8"
server_key_iv: "d8cf4bf058220098b5ce6616bcf06a8e"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.299980 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:57.321959 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:57.325136 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:57.325348 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:57.332470 5941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39933
I20260620 04:35:57.332751 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39933 every 8 connection(s)
I20260620 04:35:57.333443 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:57.336079 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:57.336570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 5941
I20260620 04:35:57.336687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:57.336966 392 external_mini_cluster.cc:1428] Setting key 8ef447063c1a701b4cbea982c444b6c2
I20260620 04:35:57.341993 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff: Bootstrap starting.
I20260620 04:35:57.344161 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:57.344969 6009 log.cc:824] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:57.346973 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff: No bootstrap required, opened a new log
I20260620 04:35:57.350492 6009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } }
I20260620 04:35:57.350704 6009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:57.350770 6009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a4a5f7fb74478d96d95a26a60517ff, State: Initialized, Role: FOLLOWER
I20260620 04:35:57.351271 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:57.338550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.98.62:39933)
0620 04:35:57.338989 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:57.339002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:57.339037 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:35:57.339657 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:57.339672 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:57.339967 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:57.340328 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:57.341135 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.342111 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.342949 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.343414 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.343586 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:57.346310 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:57.346316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:57.346336 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:57.346361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:57.348132 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.348760 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.348766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.348778 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.348906 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.349191 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.349195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.349197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.349565 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:57.349771 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:57.350055 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 04:35:57.350414 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260620 04:35:57.351236 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } }
I20260620 04:35:57.351477 6009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:57.351617 6009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:57.351743 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:57.352710 6009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } }
I20260620 04:35:57.353032 6009 leader_election.cc:302] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a4a5f7fb74478d96d95a26a60517ff; no voters:
I20260620 04:35:57.353361 6009 leader_election.cc:288] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:57.353434 6014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:57.353762 6014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [term 1 LEADER]: Becoming Leader. State: Replica: 79a4a5f7fb74478d96d95a26a60517ff, State: Running, Role: LEADER
I20260620 04:35:57.354086 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } }
I20260620 04:35:57.354718 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:57.355679 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a4a5f7fb74478d96d95a26a60517ff. Latest consensus state: current_term: 1 leader_uuid: "79a4a5f7fb74478d96d95a26a60517ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } } }
I20260620 04:35:57.355821 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:57.356629 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a4a5f7fb74478d96d95a26a60517ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a4a5f7fb74478d96d95a26a60517ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39933 } } }
I20260620 04:35:57.356721 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:57.357035 6023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:57.359445 6023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:57.365110 6023 catalog_manager.cc:1347] Generated new cluster ID: 2964b8ac6bed4fbcb171f84bd813e8d5
I20260620 04:35:57.365187 6023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:57.376873 6023 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:57.377866 6023 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:57.388475 6023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a4a5f7fb74478d96d95a26a60517ff: Generated new TSK 0
I20260620 04:35:57.389149 6023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:35:57.419962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:57.533424 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:57.533666 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:57.533710 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:57.537838 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:57.537894 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:57.537952 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:57.543087 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:57.544061 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:57.545109 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:57.554329 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.554428 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.554329 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.554560 6035 server_base.cc:1034] running on GCE node
I20260620 04:35:57.555060 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:57.555745 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:57.556937 6035 hybrid_clock.cc:648] HybridClock initialized: now 1781930157556907 us; error 49 us; skew 500 ppm
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:57.559597 6035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:35:57.560485 6035 webserver.cc:460] Webserver started at http://127.0.98.1:46569/ using document root <none> and password file <none>
I20260620 04:35:57.561043 6035 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:57.561133 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:57.561352 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:57.563319 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a1cf2d91a5c4fbab59599a02d73f782"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "d1dd72a745d5cd15871562ba23dbd9d8"
server_key_iv: "0460914053a33f8cb8f0392aa03f2e80"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.563870 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1cf2d91a5c4fbab59599a02d73f782"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "d1dd72a745d5cd15871562ba23dbd9d8"
server_key_iv: "0460914053a33f8cb8f0392aa03f2e80"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.567921 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 04:35:57.570358 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.571748 6035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:35:57.571907 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "8a1cf2d91a5c4fbab59599a02d73f782"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "d1dd72a745d5cd15871562ba23dbd9d8"
server_key_iv: "0460914053a33f8cb8f0392aa03f2e80"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.572031 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:57.592711 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:57.595942 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:57.596172 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:57.596932 6035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:57.597790 6035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:57.597862 6035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.597937 6035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:57.597988 6035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.607640 6035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41871
I20260620 04:35:57.607667 6163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41871 every 8 connection(s)
I20260620 04:35:57.608518 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:57.616705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6035
I20260620 04:35:57.616829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:57.617122 392 external_mini_cluster.cc:1428] Setting key fbf7588d6fffe73fad3f489009f1f3f2
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:57.623030 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:57.610564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41003 (local address 127.0.98.62:39933)
0620 04:35:57.610736 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:57.610740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:57.611522 (+ 782us) server_negotiation.cc:407] Connection header received
0620 04:35:57.612966 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:57.612969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:57.613023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:57.613133 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:57.614274 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.614807 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.615917 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.616149 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.616421 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:57.619106 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:57.619113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:57.619119 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:57.619149 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:57.620788 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.621439 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.621442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.621443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.621522 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.621913 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.621916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.621917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.622081 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:57.622158 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:57.622806 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 04:35:57.622913 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 04:35:57.624243 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:57.610894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39933 (local address 127.0.98.1:41003)
0620 04:35:57.611360 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:57.611401 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:57.612757 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:57.613297 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:57.613303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:57.613648 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:57.614131 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:57.614145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.614930 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.614940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:57.615793 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:57.615802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.616453 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.616457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:57.616639 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:57.617266 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:57.617282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:57.618948 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:57.620904 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:57.620912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:57.620929 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:57.621315 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:57.621630 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:57.621636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:57.621641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:57.621820 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:57.622186 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:57.622195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:57.622517 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 04:35:57.622748 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 04:35:57.623103 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260620 04:35:57.625555 6164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39933
I20260620 04:35:57.625826 6164 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:57.626314 6164 heartbeater.cc:511] Master 127.0.98.62:39933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:57.627818 5973 ts_manager.cc:194] Registered new tserver with Master: 8a1cf2d91a5c4fbab59599a02d73f782 (127.0.98.1:41871)
I20260620 04:35:57.629032 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:35:57.647863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:57.759176 6169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:57.759429 6169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:57.759472 6169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:57.763628 6169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:57.763685 6169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:57.763744 6169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:57.768962 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:57.769953 6169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:57.770912 6169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:57.780650 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.780654 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.780725 6169 server_base.cc:1034] running on GCE node
W20260620 04:35:57.780678 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.781265 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:57.781945 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:57.783159 6169 hybrid_clock.cc:648] HybridClock initialized: now 1781930157783144 us; error 38 us; skew 500 ppm
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:57.785946 6169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:35:57.786794 6169 webserver.cc:460] Webserver started at http://127.0.98.2:34749/ using document root <none> and password file <none>
I20260620 04:35:57.787302 6169 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:57.787359 6169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:57.787524 6169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:57.789567 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "56d04a94e7d142079d252897d6788fbe"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "e0cd687403fe932a61a50ea7d4dc0d96"
server_key_iv: "fdcf6c9d971922e35215dbcbc27c1b43"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.790048 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56d04a94e7d142079d252897d6788fbe"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "e0cd687403fe932a61a50ea7d4dc0d96"
server_key_iv: "fdcf6c9d971922e35215dbcbc27c1b43"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.793783 6169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:35:57.796468 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.797782 6169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:57.797943 6169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "56d04a94e7d142079d252897d6788fbe"
format_stamp: "Formatted at 2026-06-20 04:35:57 on dist-test-slave-zmf4"
server_key: "e0cd687403fe932a61a50ea7d4dc0d96"
server_key_iv: "fdcf6c9d971922e35215dbcbc27c1b43"
server_key_version: "encryptionkey@0"
I20260620 04:35:57.798091 6169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:57.813452 6169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:57.816730 6169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:57.816916 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:57.817462 6169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:57.818315 6169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:57.818367 6169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.818406 6169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:57.818423 6169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:57.827353 6169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35795
I20260620 04:35:57.827380 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35795 every 8 connection(s)
I20260620 04:35:57.828316 6169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:35:57.834213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6169
I20260620 04:35:57.834345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:35:57.834600 392 external_mini_cluster.cc:1428] Setting key cae7425e29d4b9004b8f248dfef627bc
Jun 20 04:35:57 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:57.842821 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:57.830026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33577 (local address 127.0.98.62:39933)
0620 04:35:57.830159 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:57.830163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:57.830972 (+ 809us) server_negotiation.cc:407] Connection header received
0620 04:35:57.832508 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:57.832512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:57.832570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:57.832680 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:57.833953 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.834703 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.836035 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.836324 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.836434 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:57.839176 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:57.839182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:57.839189 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:57.839221 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:57.840820 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.841351 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.841355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.841357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.841439 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:57.841780 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:57.841785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:57.841786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:57.841968 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:57.842079 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:57.842551 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 04:35:57.842663 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:35:57.843631 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:57.830353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39933 (local address 127.0.98.2:33577)
0620 04:35:57.830819 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:57.830862 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:57.832305 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:57.832849 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:57.832855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:57.833190 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:57.833794 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:57.833808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.834845 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.834850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:57.835914 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:57.835923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:57.836468 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:57.836473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:57.836638 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:57.837248 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:57.837264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:57.839003 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:57.840962 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:57.840968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:57.840983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:57.841241 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:57.841541 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:57.841544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:57.841546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:57.841681 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:57.842076 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:57.842083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:57.842321 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:35:57.842517 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 04:35:57.842836 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:35:57.844895 6298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39933
I20260620 04:35:57.845149 6298 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:57.845695 6298 heartbeater.cc:511] Master 127.0.98.62:39933 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:35:57.846822 5973 ts_manager.cc:194] Registered new tserver with Master: 56d04a94e7d142079d252897d6788fbe (127.0.98.2:35795)
I20260620 04:35:57.847654 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:35:57.863178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:57.973588 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:57.973830 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:57.973872 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:57.977946 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:57.977998 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:57.978057 6303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:35:57.982993 6303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:57.983881 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:57.984787 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:57.993870 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.993870 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:57.994099 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:57.994132 6303 server_base.cc:1034] running on GCE node
I20260620 04:35:57.994715 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:57.995337 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:57.996594 6303 hybrid_clock.cc:648] HybridClock initialized: now 1781930157996580 us; error 27 us; skew 500 ppm
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.004668 6303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:35:58.005544 6303 webserver.cc:460] Webserver started at http://127.0.98.3:42481/ using document root <none> and password file <none>
I20260620 04:35:58.006078 6303 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:58.006136 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:58.006297 6303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:58.008397 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "9abec99597914cee8c9fa05d45d10c11"
format_stamp: "Formatted at 2026-06-20 04:35:58 on dist-test-slave-zmf4"
server_key: "bf73f02c60653e38e3f2731d6b714698"
server_key_iv: "cf6d231d206c5abb6bbf855a1b2b7b44"
server_key_version: "encryptionkey@0"
I20260620 04:35:58.008920 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9abec99597914cee8c9fa05d45d10c11"
format_stamp: "Formatted at 2026-06-20 04:35:58 on dist-test-slave-zmf4"
server_key: "bf73f02c60653e38e3f2731d6b714698"
server_key_iv: "cf6d231d206c5abb6bbf855a1b2b7b44"
server_key_version: "encryptionkey@0"
I20260620 04:35:58.012635 6303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:58.015193 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:58.016736 6303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:35:58.016842 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "9abec99597914cee8c9fa05d45d10c11"
format_stamp: "Formatted at 2026-06-20 04:35:58 on dist-test-slave-zmf4"
server_key: "bf73f02c60653e38e3f2731d6b714698"
server_key_iv: "cf6d231d206c5abb6bbf855a1b2b7b44"
server_key_version: "encryptionkey@0"
I20260620 04:35:58.016929 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:58.057869 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:58.061208 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:58.061367 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:58.061904 6303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:58.062700 6303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:58.062745 6303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:58.062785 6303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:58.062801 6303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:58.071852 6303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46793
I20260620 04:35:58.071938 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46793 every 8 connection(s)
I20260620 04:35:58.072764 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:35:58.079581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6303
I20260620 04:35:58.079717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:35:58.079989 392 external_mini_cluster.cc:1428] Setting key 9559da064a4f1412c9d85937415b6cb2
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:35:58.086469 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:58.074543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60281 (local address 127.0.98.62:39933)
0620 04:35:58.074710 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.074714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.075560 (+ 846us) server_negotiation.cc:407] Connection header received
0620 04:35:58.077062 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.077066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.077125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.077230 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.078375 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.078899 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.080280 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.080499 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.080614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:58.082966 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:58.082976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:58.082979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:58.083002 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:58.084488 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.085005 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.085011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.085013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.085098 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.085434 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.085437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.085439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.085613 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:58.085720 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:58.086190 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 04:35:58.086313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 04:35:58.087320 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:58.074889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39933 (local address 127.0.98.3:60281)
0620 04:35:58.075338 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:58.075441 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:58.076843 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:58.077393 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:58.077399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:58.077779 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:58.078233 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:58.078244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.079061 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.079066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:58.080133 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:58.080142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.080646 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.080651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:58.080816 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:58.081411 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:58.081427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:58.082807 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:58.084605 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:58.084611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:58.084626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:58.084878 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:58.085193 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:58.085196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:58.085198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:58.085333 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:58.085717 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:58.085723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:58.085965 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:35:58.086152 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 04:35:58.086463 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260620 04:35:58.088613 6432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39933
I20260620 04:35:58.088851 6432 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:58.089332 6432 heartbeater.cc:511] Master 127.0.98.62:39933 requested a full tablet report, sending...
I20260620 04:35:58.090477 5973 ts_manager.cc:194] Registered new tserver with Master: 9abec99597914cee8c9fa05d45d10c11 (127.0.98.3:46793)
I20260620 04:35:58.091004 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60281
I20260620 04:35:58.093566 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/test-admin.keytab.
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.109349 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 04:35:58.112317 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:58.110827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.98.1:41871)
0620 04:35:58.111144 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.111169 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.111193 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:35:58.111242 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.111247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.111437 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.111564 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.112104 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49006: BlockingRecv error: recv got EOF from 127.0.0.1:49006 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
W20260620 04:35:58.112319 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.113545 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.114569 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:58.124619 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:58.115512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.0.98.1:41871)
0620 04:35:58.115643 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.115646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.115738 (+ 92us) server_negotiation.cc:407] Connection header received
0620 04:35:58.115794 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.115796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.115838 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.115916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.116466 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.117250 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.118103 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.118479 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.118573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:58.121022 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:58.121031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:58.121045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:58.121068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:58.122742 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.123202 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.123212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.123214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.123302 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.123599 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.123601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.123603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.123841 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:58.123999 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:58.124277 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 04:35:58.124447 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 04:35:58.128535 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:58.127479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.98.1:41871)
0620 04:35:58.127622 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.127625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.127638 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:35:58.127783 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.127788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.127834 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.127903 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.128414 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49022: BlockingRecv error: recv got EOF from 127.0.0.1:49022 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 04:35:58.128599 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.129634 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.130573 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:58.139091 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:58.131346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.0.98.1:41871)
0620 04:35:58.131471 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.131475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.131486 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:35:58.131584 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.131586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.131618 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.131682 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.132118 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.132704 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.133566 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.133798 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.133885 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:58.136005 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:58.136008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:58.136010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:58.136030 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:58.137535 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.137959 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.137961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.137963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.138047 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.138319 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.138322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.138323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.138503 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:58.138610 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:58.138782 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 04:35:58.138914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 04:35:58.141939 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:35:58.141005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.98.1:41871)
0620 04:35:58.141140 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.141144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.141238 (+ 94us) server_negotiation.cc:407] Connection header received
0620 04:35:58.141370 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.141373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.141410 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.141499 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.141849 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49040: BlockingRecv error: recv got EOF from 127.0.0.1:49040 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 04:35:58.142072 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781930128116734-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.143501 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:58.144925 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:35:58.158097 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:58.145867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.98.1:41871)
0620 04:35:58.145981 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:58.145985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:58.146289 (+ 304us) server_negotiation.cc:407] Connection header received
0620 04:35:58.146401 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:58.146406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:58.146455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:58.146557 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:58.147260 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.147945 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.149451 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:58.149911 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:58.150167 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:58.153806 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:58.153812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:58.153817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:58.153840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:58.155525 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.156323 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.156327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.156329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.156422 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:58.156743 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:58.156748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:58.156750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:58.157057 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:58.157359 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:58.157689 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:35:58.157894 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:35:58.159377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6035
I20260620 04:35:58.166061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6169
I20260620 04:35:58.172528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6303
I20260620 04:35:58.179976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 5941
2026-06-20T04:35:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2288 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[6452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[6452](info): set up 2 sockets
Jun 20 04:35:58 dist-test-slave-zmf4 krb5kdc[6452](info): commencing operation
krb5kdc: starting...
W20260620 04:35:59.229104 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:35:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:35:59.347267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:59.456053 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:59.456352 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:59.456400 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:59.461107 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:35:59.461179 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:59.461201 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:59.461308 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:35:59.461340 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:35:59.467217 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:59.468340 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:59.469357 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:59.477687 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:59.477705 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:59.477874 6465 server_base.cc:1034] running on GCE node
W20260620 04:35:59.477735 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:59.478524 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:59.479573 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:59.480731 6465 hybrid_clock.cc:648] HybridClock initialized: now 1781930159480704 us; error 39 us; skew 500 ppm
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:59.483425 6465 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:35:59.484309 6465 webserver.cc:460] Webserver started at http://127.0.98.62:42731/ using document root <none> and password file <none>
I20260620 04:35:59.484854 6465 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:59.484942 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:59.485167 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:59.487136 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "22458d73442e4504a2d0dac7c3be5c29"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "eeb1da3e00708c7a19383424e0b44b2d"
server_key_iv: "644d7df1cbdb1459f21817b8f84c0955"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.487696 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "22458d73442e4504a2d0dac7c3be5c29"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "eeb1da3e00708c7a19383424e0b44b2d"
server_key_iv: "644d7df1cbdb1459f21817b8f84c0955"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.491305 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:35:59.493716 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:59.494920 6465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:35:59.495074 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "22458d73442e4504a2d0dac7c3be5c29"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "eeb1da3e00708c7a19383424e0b44b2d"
server_key_iv: "644d7df1cbdb1459f21817b8f84c0955"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.495183 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:59.521138 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:59.524358 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:59.524569 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:59.531641 6465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36763
I20260620 04:35:59.531929 6532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36763 every 8 connection(s)
I20260620 04:35:59.532699 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:35:59.533495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6465
I20260620 04:35:59.533638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:35:59.533928 392 external_mini_cluster.cc:1428] Setting key c49bf0142a5aa65033121e0eca9e6107
I20260620 04:35:59.535701 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:35:59.542368 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Bootstrap starting.
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:35:59.545312 6533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Neither blocks nor log segments found. Creating new log.
I20260620 04:35:59.546245 6533 log.cc:824] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Log is configured to *not* fsync() on all Append() calls
I20260620 04:35:59.548647 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: No bootstrap required, opened a new log
I20260620 04:35:59.549316 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:59.535362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.0.98.62:36763)
0620 04:35:59.535771 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:59.535787 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:59.535851 (+ 64us) server_negotiation.cc:407] Connection header received
0620 04:35:59.536551 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:59.536567 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:59.536883 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:59.537186 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:59.538094 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.539076 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.540227 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.540684 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.540814 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:59.543802 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:59.543811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:59.543827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:59.543860 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:59.546048 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:59.546547 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:59.546556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:59.546562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:59.546724 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:59.547097 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:59.547100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:59.547101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:59.547612 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:59.547802 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:59.548067 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 04:35:59.548387 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260620 04:35:59.552489 6533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260620 04:35:59.552702 6533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:35:59.552745 6533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22458d73442e4504a2d0dac7c3be5c29, State: Initialized, Role: FOLLOWER
I20260620 04:35:59.553078 6533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260620 04:35:59.553186 6533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:35:59.553223 6533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:35:59.553284 6533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:35:59.554075 6533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260620 04:35:59.554353 6533 leader_election.cc:302] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22458d73442e4504a2d0dac7c3be5c29; no voters:
I20260620 04:35:59.554625 6533 leader_election.cc:288] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:35:59.554708 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:35:59.554915 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 LEADER]: Becoming Leader. State: Replica: 22458d73442e4504a2d0dac7c3be5c29, State: Running, Role: LEADER
I20260620 04:35:59.555199 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260620 04:35:59.555862 6533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:35:59.556797 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22458d73442e4504a2d0dac7c3be5c29. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } }
I20260620 04:35:59.556926 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:59.557225 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } }
I20260620 04:35:59.557318 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:35:59.558357 6547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:35:59.560913 6547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:35:59.566100 6547 catalog_manager.cc:1347] Generated new cluster ID: 9f419d9d39334c208c7d601f57d82828
I20260620 04:35:59.566166 6547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:35:59.587993 6547 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:35:59.589247 6547 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:35:59.598060 6547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Generated new TSK 0
I20260620 04:35:59.598847 6547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:35:59.631745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:59.736701 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:59.736953 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:59.737046 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:59.740931 6559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:59.741007 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:59.741107 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:35:59.746168 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:59.747087 6559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:59.748013 6559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:59.756822 6564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:59.756815 6567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:59.756822 6565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:59.757118 6559 server_base.cc:1034] running on GCE node
I20260620 04:35:59.757591 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:59.758167 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:59.759364 6559 hybrid_clock.cc:648] HybridClock initialized: now 1781930159759346 us; error 30 us; skew 500 ppm
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:59.762137 6559 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:35:59.762848 6559 webserver.cc:460] Webserver started at http://127.0.98.1:41101/ using document root <none> and password file <none>
I20260620 04:35:59.763329 6559 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:59.763386 6559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:59.763533 6559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:59.765406 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e46b13c7c9548dd9a168ca46978f838"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "1168e7f85ff3825aeb2c6ed3311b59e5"
server_key_iv: "5e12829e6cd701b2fce7d3fa47fe3f22"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.765883 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e46b13c7c9548dd9a168ca46978f838"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "1168e7f85ff3825aeb2c6ed3311b59e5"
server_key_iv: "5e12829e6cd701b2fce7d3fa47fe3f22"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.769562 6559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:59.771741 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:59.772925 6559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:35:59.773042 6559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3e46b13c7c9548dd9a168ca46978f838"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "1168e7f85ff3825aeb2c6ed3311b59e5"
server_key_iv: "5e12829e6cd701b2fce7d3fa47fe3f22"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.773141 6559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:35:59.799906 6559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:35:59.803481 6559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:35:59.803717 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:35:59.804436 6559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:35:59.805301 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:35:59.805356 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:59.805399 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:35:59.805455 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:59.815012 6559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39113
I20260620 04:35:59.815030 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39113 every 8 connection(s)
I20260620 04:35:59.815845 6559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:35:59.817946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6559
I20260620 04:35:59.818076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:35:59.818399 392 external_mini_cluster.cc:1428] Setting key 3b42cdd275d9a870c10644f91b3173cf
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:35:59.830785 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:59.817762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34799 (local address 127.0.98.62:36763)
0620 04:35:59.817919 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:35:59.817926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:35:59.818613 (+ 687us) server_negotiation.cc:407] Connection header received
0620 04:35:59.820285 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:35:59.820288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:35:59.820330 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:35:59.820398 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:35:59.821995 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.822532 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.824333 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.824574 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.824657 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:59.827404 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:35:59.827407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:35:59.827409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:35:59.827431 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:35:59.828865 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:59.829355 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:59.829358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:59.829360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:59.829443 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:35:59.829761 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:35:59.829766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:35:59.829768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:35:59.829943 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:35:59.830054 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:35:59.830499 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 04:35:59.830618 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 04:35:59.831594 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:35:59.817942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:34799)
0620 04:35:59.818458 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 04:35:59.818516 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:35:59.820045 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:35:59.820565 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:35:59.820576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:35:59.821188 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:35:59.821853 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:59.821868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.822655 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.822659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:59.824173 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:35:59.824187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:35:59.824726 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:35:59.824733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:35:59.824955 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:35:59.825568 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:35:59.825593 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:35:59.827242 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:35:59.828981 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:59.828985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:59.829003 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:59.829233 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:59.829540 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:35:59.829543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:35:59.829545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:35:59.829670 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:35:59.830064 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:35:59.830073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:35:59.830282 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 04:35:59.830458 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 04:35:59.830757 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260620 04:35:59.832810 6688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:35:59.833060 6688 heartbeater.cc:464] Registering TS with master...
I20260620 04:35:59.833675 6688 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:35:59.835402 6497 ts_manager.cc:194] Registered new tserver with Master: 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
I20260620 04:35:59.837061 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:35:59.847569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:35:59.955749 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:35:59.955977 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:35:59.956018 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:35:59.960155 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:35:59.960223 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:35:59.960281 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:35:59.965348 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:35:59.966326 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:35:59.967284 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:35:59.976430 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:59.976418 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:35:59.976464 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:35:59.976590 6693 server_base.cc:1034] running on GCE node
I20260620 04:35:59.977049 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:35:59.977648 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:35:59.978847 6693 hybrid_clock.cc:648] HybridClock initialized: now 1781930159978818 us; error 42 us; skew 500 ppm
Jun 20 04:35:59 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:35:59.981524 6693 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:35:59.982352 6693 webserver.cc:460] Webserver started at http://127.0.98.2:42005/ using document root <none> and password file <none>
I20260620 04:35:59.982832 6693 fs_manager.cc:359] Metadata directory not provided
I20260620 04:35:59.982888 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:35:59.983047 6693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:35:59.984941 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "c8adb480d0686b1d7296b1a50765ed37"
server_key_iv: "d05f27bec32858942db5e22024ec8904"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.985450 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "c8adb480d0686b1d7296b1a50765ed37"
server_key_iv: "d05f27bec32858942db5e22024ec8904"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.988914 6693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:35:59.991163 6708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:35:59.992496 6693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:35:59.992602 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "c8adb480d0686b1d7296b1a50765ed37"
server_key_iv: "d05f27bec32858942db5e22024ec8904"
server_key_version: "encryptionkey@0"
I20260620 04:35:59.992700 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.019420 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.022738 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:00.022902 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:00.023494 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:00.024363 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:00.024413 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.024452 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:00.024499 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.033852 6693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34511
I20260620 04:36:00.033900 6821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34511 every 8 connection(s)
I20260620 04:36:00.034729 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:00.043457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6693
I20260620 04:36:00.043601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:36:00.043932 392 external_mini_cluster.cc:1428] Setting key e2879eaafa42413758bc9b8f2d4fc71d
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:00.049350 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.036606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34223 (local address 127.0.98.62:36763)
0620 04:36:00.036766 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.036770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.037585 (+ 815us) server_negotiation.cc:407] Connection header received
0620 04:36:00.038910 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.038913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.038969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.039067 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.040180 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.040732 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.041827 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.042045 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.042150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.045147 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.045151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.045152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.045174 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.046927 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.047514 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.047520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.047522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.047604 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.047939 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.047944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.047946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.048129 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.048293 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.048948 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.049095 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:36:00.049969 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.036952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:34223)
0620 04:36:00.037431 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.037472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.038712 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.039237 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.039243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.039590 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.040037 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.040049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.040852 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.040856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.041711 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.041720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.042191 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.042196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.042352 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.042924 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.042940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.044961 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.047050 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.047054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.047065 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.047404 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.047699 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.047703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.047705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.047848 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.048343 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.048350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.048685 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.048858 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.049124 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260620 04:36:00.051270 6822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:36:00.051529 6822 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:00.051991 6822 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:36:00.053092 6497 ts_manager.cc:194] Registered new tserver with Master: ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:36:00.053676 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:36:00.072043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:00.181694 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:00.181975 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:00.182049 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:00.186100 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:00.186199 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:00.186295 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:00.191584 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:00.192665 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:00.193665 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:00.203389 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.203397 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.203505 6827 server_base.cc:1034] running on GCE node
W20260620 04:36:00.203397 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.204042 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:00.204685 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:00.205861 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781930160205832 us; error 26 us; skew 500 ppm
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:00.208813 6827 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:36:00.209570 6827 webserver.cc:460] Webserver started at http://127.0.98.3:35141/ using document root <none> and password file <none>
I20260620 04:36:00.210060 6827 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.210114 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.210274 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:00.212136 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "2456e20bdc3f4822aa10812bf0dcc6c5"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "d9a73a6a1266b747fb600b2c6c5775c6"
server_key_iv: "92e44f2bf91eee748c9103dc2ec38477"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.212702 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2456e20bdc3f4822aa10812bf0dcc6c5"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "d9a73a6a1266b747fb600b2c6c5775c6"
server_key_iv: "92e44f2bf91eee748c9103dc2ec38477"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.216168 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:00.218688 6842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.219925 6827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:00.220059 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "2456e20bdc3f4822aa10812bf0dcc6c5"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "d9a73a6a1266b747fb600b2c6c5775c6"
server_key_iv: "92e44f2bf91eee748c9103dc2ec38477"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.220167 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.238507 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.241627 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:00.241832 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:00.242416 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:00.243238 6827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:00.243309 6827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.243378 6827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:00.243422 6827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.252354 6827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34757
I20260620 04:36:00.252381 6955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34757 every 8 connection(s)
I20260620 04:36:00.253212 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:00.258373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6827
I20260620 04:36:00.258565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:00.258899 392 external_mini_cluster.cc:1428] Setting key f38d1040384c9d6dd14a2106467d5fec
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:00.267613 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.255121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43451 (local address 127.0.98.62:36763)
0620 04:36:00.255284 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.255293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.255988 (+ 695us) server_negotiation.cc:407] Connection header received
0620 04:36:00.257533 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.257537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.257601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.257715 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.259373 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.260100 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.261375 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.261600 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.261705 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.264112 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.264115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.264117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.264140 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.265584 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.266065 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.266068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.266069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.266157 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.266581 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.266588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.266589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.266790 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.266876 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.267383 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.267485 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:36:00.268441 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.255389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:43451)
0620 04:36:00.255844 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.255884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.257272 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.257945 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.257956 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.258497 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.259166 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.259202 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.260272 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.260275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.261259 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.261268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.261766 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.261771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.261938 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.262487 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.262512 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.263944 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.265691 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.265696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.265707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.265955 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.266273 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.266275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.266276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.266412 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.266911 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.266916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.267145 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.267313 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.267557 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260620 04:36:00.269554 6956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:36:00.269799 6956 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:00.270285 6956 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:36:00.271307 6497 ts_manager.cc:194] Registered new tserver with Master: 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3:34757)
I20260620 04:36:00.272076 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43451
I20260620 04:36:00.272729 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 04:36:00.299861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35623
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763,127.0.98.61:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:00.408921 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:00.409216 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:00.409293 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:00.413208 6961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:00.413287 6961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:00.413326 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:00.413375 6961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:00.413412 6961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:00.418816 6961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763,127.0.98.61:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
W20260620 04:36:00.419083 6961 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36763,127.0.98.61:35623'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 04:36:00.420077 6961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:00.421144 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:00.429498 6969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.429498 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.429703 6966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.429725 6961 server_base.cc:1034] running on GCE node
I20260620 04:36:00.430390 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:00.431416 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:00.432608 6961 hybrid_clock.cc:648] HybridClock initialized: now 1781930160432589 us; error 27 us; skew 500 ppm
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:00.435603 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:36:00.436553 6961 webserver.cc:460] Webserver started at http://127.0.98.61:34461/ using document root <none> and password file <none>
I20260620 04:36:00.437135 6961 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.437223 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.437440 6961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:00.439460 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/instance:
uuid: "6580f9d393b84455bff15c6e36e41894"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "c12712fd7d9611aa90ae547fe276d6cc"
server_key_iv: "3d344b86b3d036dfacfd8afee15e1aba"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.440040 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal/instance:
uuid: "6580f9d393b84455bff15c6e36e41894"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "c12712fd7d9611aa90ae547fe276d6cc"
server_key_iv: "3d344b86b3d036dfacfd8afee15e1aba"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.443822 6961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:36:00.446697 6976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.448522 6961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:00.448675 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "6580f9d393b84455bff15c6e36e41894"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "c12712fd7d9611aa90ae547fe276d6cc"
server_key_iv: "3d344b86b3d036dfacfd8afee15e1aba"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.448789 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.461807 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.465461 6961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:00.465665 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:00.473452 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35623
I20260620 04:36:00.474592 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:36:00.475471 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35623 every 8 connection(s)
I20260620 04:36:00.476272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 6961
I20260620 04:36:00.479447 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:00.483889 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:00.492442 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.478095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.0.98.61:35623)
0620 04:36:00.478622 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.478643 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.478679 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:36:00.479580 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.479603 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.480032 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.481396 (+ 1364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.481776 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.483173 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.484040 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.484591 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.484761 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.487586 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.487592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.487607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.487631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.489742 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.490151 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.490158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.490164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.490293 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.490557 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.490562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.490564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.490914 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.491122 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.491399 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.491684 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":214,"threads_started":1}
I20260620 04:36:00.494820 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.485718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.0.98.62:36763)
0620 04:36:00.485846 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.485852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.486144 (+ 292us) server_negotiation.cc:407] Connection header received
0620 04:36:00.486307 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.486311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.486359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.486444 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.487066 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.487598 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.488910 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.489283 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.489398 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.491494 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.491498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.491501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.491526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.493031 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.493561 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.493566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.493568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.493650 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.493921 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.493926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.493927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.494136 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.494241 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.494557 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.494660 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:36:00.494840 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.485635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60210)
0620 04:36:00.486028 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.486065 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.486200 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.486488 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.486493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.486832 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.486949 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.486954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.487725 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.487728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.488783 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.488794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.489396 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.489399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.489479 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.489954 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.489965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.491363 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.493135 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.493145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.493150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.493436 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.493745 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.493749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.493751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.493834 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.494248 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.494256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.494362 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.494492 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.494661 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":167,"threads_started":1}
I20260620 04:36:00.495932 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:36:00.504055 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.496426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39892)
0620 04:36:00.496525 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.496540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.496605 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.496822 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.496824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.496954 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.497067 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.497072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.498160 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.498162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.498733 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.498742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.499168 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.499171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.499248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.499643 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.499669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.500953 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.502543 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.502545 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.502547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.502757 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.503155 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.503158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.503159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.503240 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.503652 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.503654 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.503742 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.503837 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.503934 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 04:36:00.504238 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.496519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.0.98.61:35623)
0620 04:36:00.496626 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.496629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.496660 (+ 31us) server_negotiation.cc:407] Connection header received
0620 04:36:00.496687 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.496688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.496718 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.496778 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.497173 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.498042 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.498849 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.499077 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.499161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.501073 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.501075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.501077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.501096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.502429 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.502876 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.502879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.502880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.502961 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.503372 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.503375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.503376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.503545 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.503630 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.503973 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.504093 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:36:00.506965 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Bootstrap starting.
I20260620 04:36:00.509338 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:00.510035 7029 log.cc:824] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:00.511864 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: No bootstrap required, opened a new log
I20260620 04:36:00.514771 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } }
I20260620 04:36:00.514978 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:00.515069 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6580f9d393b84455bff15c6e36e41894, State: Initialized, Role: FOLLOWER
I20260620 04:36:00.515472 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } }
I20260620 04:36:00.515939 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } }
I20260620 04:36:00.516047 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:00.516650 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 04:36:00.521265 7046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:36:00.521348 7046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:36:00.522919 6961 master_runner.cc:428] Detected that this master 6580f9d393b84455bff15c6e36e41894 is joining an existing cluster
I20260620 04:36:00.522980 6961 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35623
I20260620 04:36:00.533174 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.525394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39908)
0620 04:36:00.525882 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.525891 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.526006 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.526259 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.526262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.526465 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.526587 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.526596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.527282 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.527285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.527910 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.527919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.528428 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.528431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.528537 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.529034 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.529048 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.529400 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.531457 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.531465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.531468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.531746 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.532053 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.532057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.532060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.532147 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.532657 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.532659 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.532734 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.532837 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.532972 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
I20260620 04:36:00.533157 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.525200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.98.61:35623)
0620 04:36:00.525318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.525323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.525964 (+ 641us) server_negotiation.cc:407] Connection header received
0620 04:36:00.526107 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.526110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.526153 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.526229 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.526698 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.527163 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.528020 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.528318 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.528418 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.529530 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.529534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.529537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.529561 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.531321 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.531855 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.531861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.531864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.531947 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.532277 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.532282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.532284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.532512 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.532586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.532877 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.533021 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":36}
I20260620 04:36:00.538355 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.525270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.98.62:36763)
0620 04:36:00.525376 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.525380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.525714 (+ 334us) server_negotiation.cc:407] Connection header received
0620 04:36:00.525799 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.525802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.525850 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.525929 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.526480 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.527299 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.528538 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.528875 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.528972 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.530467 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.530471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.530474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.530498 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.536804 (+ 6306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.537237 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.537241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.537243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.537321 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.537590 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.537593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.537595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.537762 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.537858 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.538115 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.538222 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:36:00.538371 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.525186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60216)
0620 04:36:00.525618 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.525637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.525718 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.525966 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.525969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.526192 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.526359 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.526367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.527430 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.527433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.528402 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.528413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.529101 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.529105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.529192 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.529834 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.529848 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.530325 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.536919 (+ 6594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.536925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.536927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.537140 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.537416 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.537419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.537421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.537502 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.537859 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.537862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.537951 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.538054 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.538190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":20,"thread_start_us":105,"threads_started":1}
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:36:00.554107 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.542536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.0.98.61:35623)
0620 04:36:00.542647 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.542651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.544626 (+ 1975us) server_negotiation.cc:407] Connection header received
0620 04:36:00.544817 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.544820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.544866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.544951 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.545713 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.546478 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.547554 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.547782 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.547883 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.550340 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.550343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.550344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.550364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.551846 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.552556 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.552559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.552560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.552690 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.553110 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.553115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.553117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.553383 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.553486 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.553897 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.553989 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:36:00.554162 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.542521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39912)
0620 04:36:00.544487 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.544538 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.544692 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.544978 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.544984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.545440 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.545562 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.545585 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.546633 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.546639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.547441 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.547449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.547885 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.547889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.547993 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.548531 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.548543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.550177 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.552082 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.552091 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.552097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.552441 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.552831 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.552834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.552836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.552929 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.553501 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.553507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.553635 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.553781 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.553977 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"thread_start_us":84,"threads_started":1}
I20260620 04:36:00.555783 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35623: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6580f9d393b84455bff15c6e36e41894"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35623
}
attrs {
promote: true
}
}
dest_uuid: "22458d73442e4504a2d0dac7c3be5c29"
cas_config_opid_index: -1
I20260620 04:36:00.556507 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 04:36:00.558182 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6580f9d393b84455bff15c6e36e41894 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.559046 7003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 04:36:00.561228 7003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 04:36:00.561923 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:00.562719 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35623
I20260620 04:36:00.563130 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.563242 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:00.567025 7003 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6580f9d393b84455bff15c6e36e41894 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.567478 6961 master.cc:537] Master@127.0.98.61:35623 shutting down...
I20260620 04:36:00.568224 7062 mvcc.cc:205] Tried to move back new op lower bound from 7298785933709869056 to 7298785933538828288. Current Snapshot: MvccSnapshot[applied={T|T < 7298785933573591040}]
I20260620 04:36:00.571650 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22458d73442e4504a2d0dac7c3be5c29. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.571779 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.571892 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.571979 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.572084 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.572152 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.573788 6961 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 04:36:00.574071 6961 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 04:36:00.574172 6961 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 04:36:00.583168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44907
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:36:00.590131 6961 master.cc:552] Master@127.0.98.61:35623 shutdown complete.
I20260620 04:36:00.590198 6961 master_runner.cc:305] Clearing existing system tablet
I20260620 04:36:00.590437 6961 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.590492 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.591776 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 04:36:00.592707 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.592952 6961 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.593072 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:00.593104 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "6580f9d393b84455bff15c6e36e41894"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "c12712fd7d9611aa90ae547fe276d6cc"
server_key_iv: "3d344b86b3d036dfacfd8afee15e1aba"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.594853 6961 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:00.597247 6961 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:36:00.597333 6961 log.cc:1196] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 04:36:00.597612 6961 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Deleting consensus metadata
I20260620 04:36:00.597777 6961 master_runner.cc:315] Copying system tablet from 127.0.98.62:36763
I20260620 04:36:00.598901 6961 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36763
I20260620 04:36:00.608476 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.599473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60226)
0620 04:36:00.600384 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.600401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.600525 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.600800 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.600803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.601027 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.601159 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.601166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.602238 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.602243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.603185 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.603195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.603838 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.603842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.603946 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.604546 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.604563 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.604989 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.606884 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.606887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.606889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.607130 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.607410 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.607413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.607414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.607510 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.607908 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.607911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.608008 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.608113 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.608280 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":84,"threads_started":1}
I20260620 04:36:00.608572 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.599505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.98.62:36763)
0620 04:36:00.599636 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.599642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.600483 (+ 841us) server_negotiation.cc:407] Connection header received
0620 04:36:00.600612 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.600615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.600668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.600760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.601279 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.602078 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.603376 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.603719 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.603821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.605135 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.605146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.605149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.605174 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.606780 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.607236 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.607239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.607241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.607325 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.607602 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.607605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.607607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.607815 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.607913 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.608287 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.608412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:36:00.609422 6517 tablet_copy_service.cc:140] P 22458d73442e4504a2d0dac7c3be5c29: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6580f9d393b84455bff15c6e36e41894 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60226)
I20260620 04:36:00.609614 6517 tablet_copy_service.cc:161] P 22458d73442e4504a2d0dac7c3be5c29: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6580f9d393b84455bff15c6e36e41894 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60226: session id = 6580f9d393b84455bff15c6e36e41894-00000000000000000000000000000000
I20260620 04:36:00.611245 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:36:00.612854 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:36:00.615061 6961 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: tablet copy: Starting download of 0 data blocks...
I20260620 04:36:00.615259 6961 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: tablet copy: Starting download of 1 WAL segments...
I20260620 04:36:00.616947 6961 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:36:00.618701 6517 tablet_copy_service.cc:342] P 22458d73442e4504a2d0dac7c3be5c29: Request end of tablet copy session 6580f9d393b84455bff15c6e36e41894-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60226
I20260620 04:36:00.618817 6517 tablet_copy_service.cc:433] P 22458d73442e4504a2d0dac7c3be5c29: ending tablet copy session 6580f9d393b84455bff15c6e36e41894-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6580f9d393b84455bff15c6e36e41894
I20260620 04:36:00.626837 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:00.628458 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.628490 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.628729 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.629091 6961 server_base.cc:1034] running on GCE node
I20260620 04:36:00.629251 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:00.629519 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:00.630654 6961 hybrid_clock.cc:648] HybridClock initialized: now 1781930160630643 us; error 29 us; skew 500 ppm
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:00.632272 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:36:00.632704 6961 webserver.cc:460] Webserver started at http://127.0.98.61:38743/ using document root <none> and password file <none>
I20260620 04:36:00.632884 6961 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.632925 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.634212 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:00.635179 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.635461 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:00.635561 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "6580f9d393b84455bff15c6e36e41894"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "c12712fd7d9611aa90ae547fe276d6cc"
server_key_iv: "3d344b86b3d036dfacfd8afee15e1aba"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.635658 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.649603 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.651733 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:00.656893 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35623
I20260620 04:36:00.656917 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35623 every 8 connection(s)
I20260620 04:36:00.657543 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:36:00.659474 7141 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:36:00.660598 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Bootstrap starting.
I20260620 04:36:00.668475 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:36:00.668799 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: Bootstrap complete.
I20260620 04:36:00.669401 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } }
I20260620 04:36:00.669538 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6580f9d393b84455bff15c6e36e41894, State: Initialized, Role: LEARNER
I20260620 04:36:00.669649 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } }
I20260620 04:36:00.669991 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.670128 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:00.670228 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.673516 7155 catalog_manager.cc:1259] Loaded cluster ID: 9f419d9d39334c208c7d601f57d82828
I20260620 04:36:00.673631 7155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: loading cluster ID for follower catalog manager: success
I20260620 04:36:00.675688 7155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: acquiring CA information for follower catalog manager: success
I20260620 04:36:00.677134 7155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 04:36:00.714537 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:00.714792 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:00.714833 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:00.718863 7069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:00.718916 7069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:00.718935 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:00.718952 7069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:00.718967 7069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:00.724094 7069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:00.725108 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:00.726045 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:00.734270 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.734282 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.734282 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.734575 7069 server_base.cc:1034] running on GCE node
I20260620 04:36:00.735129 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:00.736310 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:00.737520 7069 hybrid_clock.cc:648] HybridClock initialized: now 1781930160737490 us; error 44 us; skew 500 ppm
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:00.740460 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:36:00.741324 7069 webserver.cc:460] Webserver started at http://127.0.98.60:33191/ using document root <none> and password file <none>
I20260620 04:36:00.741861 7069 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.741921 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.742121 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:00.744124 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/instance:
uuid: "916d41cbd01542268848383252e2a3a8"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "7d39a377331e952e67823367e8343868"
server_key_iv: "1776de20eab826dcf2b25cb914cbff98"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.744735 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal/instance:
uuid: "916d41cbd01542268848383252e2a3a8"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "7d39a377331e952e67823367e8343868"
server_key_iv: "1776de20eab826dcf2b25cb914cbff98"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.748417 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:00.750919 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.752172 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:00.752368 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "916d41cbd01542268848383252e2a3a8"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "7d39a377331e952e67823367e8343868"
server_key_iv: "1776de20eab826dcf2b25cb914cbff98"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.752480 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.766114 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.769193 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:00.769356 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:00.776592 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44907
I20260620 04:36:00.776865 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44907 every 8 connection(s)
I20260620 04:36:00.777710 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:36:00.779389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7069
I20260620 04:36:00.781633 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:00.785791 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:00.794986 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.780871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.60:44907)
0620 04:36:00.781321 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.781336 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.781369 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:36:00.782049 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.782072 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.782378 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.782642 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.783505 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.784831 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.786004 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.786557 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.786680 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.789777 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.789783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.789791 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.789816 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.791770 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.792444 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.792450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.792461 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.792570 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.792897 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.792900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.792902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.793405 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.793577 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.793888 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.794257 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
I20260620 04:36:00.798043 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.787755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.0.98.62:36763)
0620 04:36:00.787910 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.787915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.788094 (+ 179us) server_negotiation.cc:407] Connection header received
0620 04:36:00.788254 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.788257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.788317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.788419 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.789305 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.789986 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.791306 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.791539 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.791626 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.794642 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.794646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.794648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.794672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.796333 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.796768 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.796773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.796775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.796854 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.797126 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.797131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.797133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.797347 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.797453 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.797770 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.797883 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:36:00.798087 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.787612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60234)
0620 04:36:00.787964 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.788010 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.788136 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.788523 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.788530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.788977 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.789174 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.789185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.790131 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.790136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.791160 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.791175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.791658 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.791663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.791781 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.792766 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.792777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.794501 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.796432 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.796441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.796446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.796670 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.796947 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.796950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.796952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.797041 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.797464 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.797472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.797585 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.797720 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.797908 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":122,"threads_started":1}
I20260620 04:36:00.799014 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:36:00.807144 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.799577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39914)
0620 04:36:00.799769 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.799785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.799860 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.800254 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.800256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.800384 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.800466 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.800471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.801465 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.801467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.801988 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.801995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.802476 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.802479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.802582 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.802961 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.802972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.804293 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.805873 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.805877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.805879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.806069 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.806351 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.806353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.806354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.806428 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.806778 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.806780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.806859 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.806942 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.807034 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 04:36:00.807205 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.799690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.0.98.61:35623)
0620 04:36:00.799978 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.799983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.800000 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:36:00.800044 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.800047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.800095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.800207 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.800593 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.801336 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.802103 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.802361 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.802446 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.804418 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.804421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.804423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.804444 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.805766 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.806171 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.806174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.806175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.806262 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.806524 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.806527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.806528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.806695 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.806773 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.806990 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.807081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260620 04:36:00.808012 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:36:00.815842 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.808465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:59430)
0620 04:36:00.808561 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.808572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.808628 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.808827 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.808829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.808945 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.809007 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.809011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.810038 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.810041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.810562 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.810569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.811091 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.811093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.811154 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.811547 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.811559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.812787 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.814388 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.814391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.814392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.814593 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.814935 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.814938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.814940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.815020 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.815358 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.815361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.815436 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.815547 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.815678 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:36:00.815851 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.808553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.98.60:44907)
0620 04:36:00.808675 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.808678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.808688 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:36:00.808714 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.808716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.808745 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.808807 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.809098 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.809932 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.810671 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.810922 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.811018 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.812921 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.812924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.812926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.812945 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.814283 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.814700 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.814703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.814704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.814794 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.815106 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.815111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.815113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.815262 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.815358 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.815588 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.815731 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":40}
I20260620 04:36:00.818500 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Bootstrap starting.
I20260620 04:36:00.820667 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:00.821373 7223 log.cc:824] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:00.823287 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: No bootstrap required, opened a new log
I20260620 04:36:00.826282 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260620 04:36:00.826473 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:00.826534 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916d41cbd01542268848383252e2a3a8, State: Initialized, Role: FOLLOWER
I20260620 04:36:00.826982 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260620 04:36:00.827468 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260620 04:36:00.827610 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:00.828152 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 04:36:00.833004 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:36:00.833097 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:36:00.836130 7069 master_runner.cc:428] Detected that this master 916d41cbd01542268848383252e2a3a8 is joining an existing cluster
I20260620 04:36:00.836223 7069 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44907
I20260620 04:36:00.839902 6688 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260620 04:36:00.847352 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.838360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.98.61:35623)
0620 04:36:00.838488 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.838494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.838870 (+ 376us) server_negotiation.cc:407] Connection header received
0620 04:36:00.838992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.838998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.839046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.839113 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.839893 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.840627 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.841801 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.842143 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.842217 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.843653 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.843657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.843661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.843944 (+ 283us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.845719 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.846096 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.846101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.846104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.846182 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.846445 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.846448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.846450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.846679 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.846772 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.847075 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.847201 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:36:00.847563 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.838487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39930)
0620 04:36:00.838788 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.838801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.838901 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.839214 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.839217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.839539 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.839761 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.839768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.840768 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.840771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.841690 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.841698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.842373 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.842376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.842459 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.843013 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.843029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.843502 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.845806 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.845812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.845814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.846005 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.846273 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.846276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.846278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.846357 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.846816 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.846819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.846914 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.847027 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.847150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":62,"spinlock_wait_cycles":6016,"thread_start_us":94,"threads_started":1}
I20260620 04:36:00.848417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6559
I20260620 04:36:00.848657 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.838286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.98.62:36763)
0620 04:36:00.838786 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.838792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.839484 (+ 692us) server_negotiation.cc:407] Connection header received
0620 04:36:00.839653 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.839656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.839706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.839757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.840417 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.841208 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.842864 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.843216 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.843291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.844552 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.844557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.844562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.844590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.846491 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.846908 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.846911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.846913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.846998 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.847314 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.847317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.847319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.847513 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.847578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.848429 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.848535 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260620 04:36:00.850004 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.839072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60238)
0620 04:36:00.839403 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.839416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.839543 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.839896 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.839899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.840128 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.840276 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.840284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.841347 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.841351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.842723 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.842736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.843443 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.843447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.843536 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.843958 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.843972 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.844392 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.846589 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.846592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.846594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.846819 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.847116 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.847120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.847122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.847212 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.848107 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.848110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.848230 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.849673 (+ 1443us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.849821 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260620 04:36:00.854676 7229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:00.854840 7229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260620 04:36:00.855948 7229 leader_election.cc:288] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763), 6580f9d393b84455bff15c6e36e41894 (127.0.98.61:35623)
I20260620 04:36:00.856647 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "916d41cbd01542268848383252e2a3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6580f9d393b84455bff15c6e36e41894" is_pre_election: true
I20260620 04:36:00.856899 7115 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Handling vote request from an unknown peer 916d41cbd01542268848383252e2a3a8
I20260620 04:36:00.857046 7115 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 916d41cbd01542268848383252e2a3a8 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 04:36:00.857362 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "916d41cbd01542268848383252e2a3a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22458d73442e4504a2d0dac7c3be5c29" is_pre_election: true
I20260620 04:36:00.858197 7173 leader_election.cc:302] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 916d41cbd01542268848383252e2a3a8; no voters: 22458d73442e4504a2d0dac7c3be5c29, 6580f9d393b84455bff15c6e36e41894
I20260620 04:36:00.858455 7229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:00.858764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39113
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=41101
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:36:00.877666 7229 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 04:36:00.882318 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.854514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:59450)
0620 04:36:00.854682 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.854698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.854778 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.855135 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.855139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.855321 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.855419 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.855425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.856992 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.856996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.857958 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.857970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.858585 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.858590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.858702 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.859296 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.859311 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.860910 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.878780 (+ 17870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.878784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.878787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.879351 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.879924 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.879927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.879929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.880026 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.881749 (+ 1723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.881752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.881857 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.881957 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.882140 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 04:36:00.882324 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.838390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:59434)
0620 04:36:00.838651 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.838667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.838755 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.839214 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.839217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.839543 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.841282 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.841292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.842223 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.842226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.843095 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.843105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.843734 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.843738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.843845 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.877956 (+ 34111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.877973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.878616 (+ 643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.880575 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.880578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.880580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.880788 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.881037 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.881039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.881041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.881115 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.881825 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.881827 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.881895 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.881979 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.882114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":69,"spinlock_wait_cycles":3968,"thread_start_us":108,"threads_started":1}
I20260620 04:36:00.882949 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.838870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.0.98.60:44907)
0620 04:36:00.838970 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.838974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.838984 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:36:00.839010 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.839013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.839046 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.839113 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.841392 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.842094 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.843230 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.843529 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.843609 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.878787 (+ 35178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.878790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.878793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.878813 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.880468 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.880884 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.880886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.880888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.880953 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.881227 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.881230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.881231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.881703 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.881793 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.882667 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.882781 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":39}
I20260620 04:36:00.883139 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.854568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.0.98.60:44907)
0620 04:36:00.854836 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.854845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.854860 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:00.854925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.854930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.855004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.855096 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.855542 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.856672 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.858067 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.858444 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.859160 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.861088 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.861092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.861095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.861120 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.878591 (+ 17471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.879472 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.879476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.879479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.879809 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.880141 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.880144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.880146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.881606 (+ 1460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.881704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.882678 (+ 974us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.882856 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260620 04:36:00.884009 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44907: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "916d41cbd01542268848383252e2a3a8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44907
}
attrs {
promote: true
}
}
dest_uuid: "22458d73442e4504a2d0dac7c3be5c29"
cas_config_opid_index: 5
I20260620 04:36:00.885020 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:00.887236 7197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:36:00.887213 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 916d41cbd01542268848383252e2a3a8 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.887943 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:00.889372 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44907
I20260620 04:36:00.889567 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.889669 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:00.892589 7069 master.cc:537] Master@127.0.98.60:44907 shutting down...
I20260620 04:36:00.894594 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.886653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39944)
0620 04:36:00.889966 (+ 3313us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.889982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.890076 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.890512 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.890515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.890531 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:00.890619 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.890624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.891662 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.891665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.892960 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.892971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.894163 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.894167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.894259 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.894306 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.894379 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.894456 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3235}
I20260620 04:36:00.894637 7197 raft_consensus.cc:858] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:00.895005 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.886949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.98.61:35623)
0620 04:36:00.887096 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.887101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.890170 (+ 3069us) server_negotiation.cc:407] Connection header received
0620 04:36:00.890233 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.890236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.890301 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.890381 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:00.890732 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.891530 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.893130 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.893768 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.893869 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.894016 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.894812 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.894890 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 04:36:00.895289 7197 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:00.895489 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:36:00.895907 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 04:36:00.897010 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 916d41cbd01542268848383252e2a3a8 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.897168 7197 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6580f9d393b84455bff15c6e36e41894 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.898705 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22458d73442e4504a2d0dac7c3be5c29. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.898818 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.898909 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.898983 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:00.899907 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.900014 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 04:36:00.900107 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.900163 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 04:36:00.900416 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22458d73442e4504a2d0dac7c3be5c29. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } }
I20260620 04:36:00.900583 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 04:36:00.903682 7251 mvcc.cc:205] Tried to move back new op lower bound from 7298785933709869056 to 7298785933538828288. Current Snapshot: MvccSnapshot[applied={T|T < 7298785933573591040}]
I20260620 04:36:00.906332 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: true } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.906479 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:00.907133 7069 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260620 04:36:00.907352 6482 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 04:36:00.907431 7069 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 04:36:00.907528 7069 tablet_replica.cc:330] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: stopping tablet replica
I20260620 04:36:00.907656 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: attempting to promote NON_VOTER 6580f9d393b84455bff15c6e36e41894 to VOTER
I20260620 04:36:00.908083 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:00.909293 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 LEARNER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 04:36:00.909750 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 04:36:00.909883 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 -> Peer 916d41cbd01542268848383252e2a3a8 (127.0.98.60:44907): Couldn't send request to peer 916d41cbd01542268848383252e2a3a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 04:36:00.911499 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6580f9d393b84455bff15c6e36e41894 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.913100 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.913206 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:00.913331 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.913399 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:00.914036 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6580f9d393b84455bff15c6e36e41894 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.915762 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:00.915880 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:00.920799 7069 master.cc:552] Master@127.0.98.60:44907 shutdown complete.
I20260620 04:36:00.920859 7069 master_runner.cc:305] Clearing existing system tablet
I20260620 04:36:00.921115 7069 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.921178 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.922533 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 04:36:00.923499 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.923751 7069 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.924041 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 04:36:00.924131 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "916d41cbd01542268848383252e2a3a8"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "7d39a377331e952e67823367e8343868"
server_key_iv: "1776de20eab826dcf2b25cb914cbff98"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.926244 7069 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:00.928804 7069 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:36:00.928956 7069 log.cc:1196] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 04:36:00.929358 7069 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Deleting consensus metadata
I20260620 04:36:00.929698 7069 master_runner.cc:315] Copying system tablet from 127.0.98.62:36763
I20260620 04:36:00.930864 7069 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36763
I20260620 04:36:00.944736 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.932665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.0.98.62:36763)
0620 04:36:00.932804 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:00.932809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:00.936068 (+ 3259us) server_negotiation.cc:407] Connection header received
0620 04:36:00.936316 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:00.936320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:00.936376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:00.936484 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:00.937127 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.937943 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.939137 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.939468 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.939537 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.941198 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:00.941208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:00.941212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:00.941250 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:00.942942 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.943441 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.943445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.943447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.943525 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:00.943806 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:00.943809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:00.943811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:00.944007 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:00.944074 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:00.944451 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 04:36:00.944567 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:36:00.945463 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:00.932541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60250)
0620 04:36:00.935938 (+ 3397us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:00.935956 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:00.936139 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:00.936529 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:00.936532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:00.936795 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:00.936955 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.936963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.938084 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.938087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.938990 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:00.939000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:00.939697 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:00.939700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:00.939797 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:00.940437 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:00.940452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:00.940866 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:00.943076 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.943079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.943081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.943333 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.943628 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:00.943631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:00.943633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:00.943711 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:00.944157 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:00.944161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:00.944272 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 04:36:00.945135 (+ 863us) client_negotiation.cc:240] Negotiation successful
0620 04:36:00.945299 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3313,"thread_start_us":91,"threads_started":1}
I20260620 04:36:00.946049 6517 tablet_copy_service.cc:140] P 22458d73442e4504a2d0dac7c3be5c29: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 916d41cbd01542268848383252e2a3a8 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60250)
I20260620 04:36:00.946159 6517 tablet_copy_service.cc:161] P 22458d73442e4504a2d0dac7c3be5c29: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 916d41cbd01542268848383252e2a3a8 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60250: session id = 916d41cbd01542268848383252e2a3a8-00000000000000000000000000000000
I20260620 04:36:00.947216 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:36:00.948513 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:36:00.950774 7069 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: tablet copy: Starting download of 0 data blocks...
I20260620 04:36:00.950954 7069 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: tablet copy: Starting download of 1 WAL segments...
I20260620 04:36:00.952555 7069 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:36:00.954319 6517 tablet_copy_service.cc:342] P 22458d73442e4504a2d0dac7c3be5c29: Request end of tablet copy session 916d41cbd01542268848383252e2a3a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60250
I20260620 04:36:00.954403 6517 tablet_copy_service.cc:433] P 22458d73442e4504a2d0dac7c3be5c29: ending tablet copy session 916d41cbd01542268848383252e2a3a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 916d41cbd01542268848383252e2a3a8
I20260620 04:36:00.963316 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:00.965023 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:00.965281 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.965523 7069 server_base.cc:1034] running on GCE node
W20260620 04:36:00.966272 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:00.966496 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:00.966816 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:00.967970 7069 hybrid_clock.cc:648] HybridClock initialized: now 1781930160967952 us; error 43 us; skew 500 ppm
Jun 20 04:36:00 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:00.969969 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:36:00.970507 7069 webserver.cc:460] Webserver started at http://127.0.98.60:33937/ using document root <none> and password file <none>
I20260620 04:36:00.970706 7069 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:00.970752 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:00.972136 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:00.973188 7279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:00.973562 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:00.973655 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "916d41cbd01542268848383252e2a3a8"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "7d39a377331e952e67823367e8343868"
server_key_iv: "1776de20eab826dcf2b25cb914cbff98"
server_key_version: "encryptionkey@0"
I20260620 04:36:00.973743 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:00.994352 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:00.996721 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:01.001519 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44907
I20260620 04:36:01.001531 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44907 every 8 connection(s)
I20260620 04:36:01.002225 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:36:01.004312 7332 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:36:01.005188 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Bootstrap starting.
I20260620 04:36:01.011938 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:36:01.012305 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Bootstrap complete.
I20260620 04:36:01.012769 7332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:01.012890 7332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 916d41cbd01542268848383252e2a3a8, State: Initialized, Role: LEARNER
I20260620 04:36:01.013002 7332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } }
I20260620 04:36:01.013327 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:01.013487 7332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:01.013494 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: LEARNER
W20260620 04:36:01.015724 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:01.016256 7250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:01.016335 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:36:01.017278 7349 catalog_manager.cc:1259] Loaded cluster ID: 9f419d9d39334c208c7d601f57d82828
I20260620 04:36:01.017346 7349 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: loading cluster ID for follower catalog manager: success
I20260620 04:36:01.018850 7349 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: acquiring CA information for follower catalog manager: success
I20260620 04:36:01.020027 7349 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 04:36:01.022683 7250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:01.022763 7250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:01.022850 7250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:01.031368 7250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41101
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:01.032727 7250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:01.033964 7250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:01.044904 7351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:01.044938 7250 server_base.cc:1034] running on GCE node
W20260620 04:36:01.044931 7352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:01.044904 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:01.045558 7250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:01.046214 7250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:01.047422 7250 hybrid_clock.cc:648] HybridClock initialized: now 1781930161047390 us; error 46 us; skew 500 ppm
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:01.050084 7250 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:36:01.050889 7250 webserver.cc:460] Webserver started at http://127.0.98.1:41101/ using document root <none> and password file <none>
I20260620 04:36:01.051417 7250 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:01.051527 7250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:01.055308 7250 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:01.055681 6822 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260620 04:36:01.057955 7361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.059244 7250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:36:01.059391 7250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3e46b13c7c9548dd9a168ca46978f838"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "1168e7f85ff3825aeb2c6ed3311b59e5"
server_key_iv: "5e12829e6cd701b2fce7d3fa47fe3f22"
server_key_version: "encryptionkey@0"
I20260620 04:36:01.059918 7250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:01.081172 7250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:01.084704 7250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:01.084923 7250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:01.085534 7250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:01.086498 7250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:01.086578 7250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.086660 7250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:01.086701 7250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.096791 7250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39113
I20260620 04:36:01.096827 7474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39113 every 8 connection(s)
I20260620 04:36:01.097821 7250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:01.101140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7250
I20260620 04:36:01.101328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6693
I20260620 04:36:01.108740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34511
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=42005
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:01.116362 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.100743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51987 (local address 127.0.98.62:36763)
0620 04:36:01.100853 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.100857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.102661 (+ 1804us) server_negotiation.cc:407] Connection header received
0620 04:36:01.102846 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.102853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.102911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.102968 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.104857 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.105669 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.107784 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108128 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.112694 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.112699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.112702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.112731 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.114623 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.115068 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.115072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.115074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.115154 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.115443 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.115446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.115448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.115673 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.115736 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.116068 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.116181 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:36:01.117048 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.100022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.98.1:56351)
0620 04:36:01.101130 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.101156 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.102561 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.103168 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.103180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.103994 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.104685 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.104706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.105851 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.105855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.107645 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.107656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108440 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.108967 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.109565 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.109581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.111756 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.113852 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.113857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.113873 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.114141 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.114420 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.114424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.114426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.114573 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.115078 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.115085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.115408 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.115606 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.115960 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":617,"spinlock_wait_cycles":17792,"thread_start_us":747,"threads_started":1}
I20260620 04:36:01.117383 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.100471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:51987)
0620 04:36:01.102565 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.102582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.102732 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.103167 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.103180 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.103873 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.104685 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.104706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.105851 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.105855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.107645 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.107656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108440 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.108821 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.110033 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.110048 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.112518 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.114733 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.114736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.114738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.114954 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.115260 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.115263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.115265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.115342 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.115805 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.115808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.115885 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.116515 (+ 630us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.116746 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":1190,"spinlock_wait_cycles":822400,"thread_start_us":454,"threads_started":1}
I20260620 04:36:01.118049 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.099655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56351 (local address 127.0.98.61:35623)
0620 04:36:01.099780 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.099785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.101234 (+ 1449us) server_negotiation.cc:407] Connection header received
0620 04:36:01.102689 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.102696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.102766 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.102825 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.104857 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.105669 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.107784 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108136 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108227 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.111960 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.111964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.111967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.111997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.113746 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.114249 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.114252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.114254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.114344 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.114673 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.114676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.114678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.114913 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.115016 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.117739 (+ 2723us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.117866 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:36:01.118391 7476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:36:01.118631 7476 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.118916 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35623
I20260620 04:36:01.118995 7477 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.119591 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.099575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33551 (local address 127.0.98.60:44907)
0620 04:36:01.099820 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.099824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.101896 (+ 2072us) server_negotiation.cc:407] Connection header received
0620 04:36:01.103381 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.103387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.103452 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.103556 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.104853 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.106059 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.107947 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108267 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108375 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.112574 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.112583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.112590 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.112646 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.116164 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.117059 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.117065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.117069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.117606 (+ 537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.117960 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.117965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.117967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.118438 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.118591 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.119147 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.119297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260620 04:36:01.119933 7476 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:36:01.119935 7477 heartbeater.cc:511] Master 127.0.98.61:35623 requested a full tablet report, sending...
I20260620 04:36:01.121151 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.100022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.98.1:33551)
0620 04:36:01.100584 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.100649 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.102268 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.103698 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.103704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.104003 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.104687 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.104707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.106197 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.106202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.107621 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.107635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.108501 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.108508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.108783 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.109412 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.109439 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.112118 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.116321 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.116327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.116331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.116911 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.117736 (+ 825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.117740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.117742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.117842 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.118655 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.118658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.118825 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.118931 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.120887 (+ 1956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":319,"spinlock_wait_cycles":9600,"thread_start_us":163,"threads_started":1}
I20260620 04:36:01.121335 6496 ts_manager.cc:194] Re-registered known tserver with Master: 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
I20260620 04:36:01.121971 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51987
I20260620 04:36:01.123199 7105 ts_manager.cc:194] Registered new tserver with Master: 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
I20260620 04:36:01.123756 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44907
I20260620 04:36:01.123834 7475 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.124085 7475 heartbeater.cc:511] Master 127.0.98.60:44907 requested a full tablet report, sending...
I20260620 04:36:01.125154 7296 ts_manager.cc:194] Registered new tserver with Master: 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
W20260620 04:36:01.229985 7483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:01.230249 7483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:01.230294 7483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:01.234474 7483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:01.234534 7483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:01.234594 7483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:01.240000 7483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42005
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:01.241150 7483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:01.242121 7483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:36:01.251907 7483 server_base.cc:1034] running on GCE node
W20260620 04:36:01.251871 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:01.251958 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:01.251871 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:01.252662 7483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:01.253295 7483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:01.254514 7483 hybrid_clock.cc:648] HybridClock initialized: now 1781930161254461 us; error 75 us; skew 500 ppm
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:01.257540 7483 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:36:01.258503 7483 webserver.cc:460] Webserver started at http://127.0.98.2:42005/ using document root <none> and password file <none>
I20260620 04:36:01.259047 7483 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:01.259119 7483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:01.262742 7483 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 04:36:01.265092 7498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.266321 7483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:36:01.266448 7483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
format_stamp: "Formatted at 2026-06-20 04:35:59 on dist-test-slave-zmf4"
server_key: "c8adb480d0686b1d7296b1a50765ed37"
server_key_iv: "d05f27bec32858942db5e22024ec8904"
server_key_version: "encryptionkey@0"
I20260620 04:36:01.267182 7483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:01.274478 6956 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260620 04:36:01.290664 7483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:01.293892 7483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:01.294111 7483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:01.294703 7483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:01.295547 7483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:01.295612 7483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.295699 7483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:01.295742 7483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.304853 7483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34511
I20260620 04:36:01.304921 7611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34511 every 8 connection(s)
I20260620 04:36:01.305857 7483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:01.307325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7483
I20260620 04:36:01.307503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6827
I20260620 04:36:01.314996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34757
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=35141
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:36:01.325346 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42927 (local address 127.0.98.62:36763)
0620 04:36:01.308563 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.308570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.310278 (+ 1708us) server_negotiation.cc:407] Connection header received
0620 04:36:01.311348 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.311352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.311404 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.311459 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.313466 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314285 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.316364 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.316755 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.316839 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.320268 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.320273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.320278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.320307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.322737 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323346 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.323350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.323352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.323448 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323977 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.323984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.323987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.324275 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.324356 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.325051 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.325179 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:36:01.325346 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50709 (local address 127.0.98.60:44907)
0620 04:36:01.309215 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.309221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.310761 (+ 1540us) server_negotiation.cc:407] Connection header received
0620 04:36:01.311353 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.311360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.311427 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.311484 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.313464 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314347 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.316344 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.316721 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.316810 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.321411 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.321418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.321423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.321456 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.323241 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323838 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.323842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.323844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.323934 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.324264 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.324267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.324269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.324473 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.324550 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.325051 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.325179 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260620 04:36:01.325462 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53539 (local address 127.0.98.61:35623)
0620 04:36:01.309058 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.309063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.310236 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 04:36:01.311348 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.311352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.311404 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.311459 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.313466 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314305 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.316848 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.317190 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.317271 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.320753 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.320759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.320764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.320795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.322576 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323342 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.323347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.323351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.323450 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.324005 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.324011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.324015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.324398 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.324503 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.325077 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.325234 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 04:36:01.327061 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.98.2:53539)
0620 04:36:01.310061 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.310120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.311109 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.311842 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.311851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.312448 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.313305 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.313320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314500 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.314503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.316728 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.316737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.317435 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.317438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.317544 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.318377 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.318395 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.320551 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.322717 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.322724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.322744 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.323214 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.323571 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.323581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.323868 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.324574 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.324579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.324751 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.326236 (+ 1485us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.326421 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1710,"mutex_wait_us":497,"spinlock_wait_cycles":32384,"thread_start_us":123,"threads_started":1}
I20260620 04:36:01.327060 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:42927)
0620 04:36:01.310061 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.310120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.311109 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.311842 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.311851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.312466 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.313307 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.313320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314472 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.314475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.316186 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.316228 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.317028 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.317031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.317229 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.318161 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.318189 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.320026 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.322872 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.322875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.322877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.323227 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.323561 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.323570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.323861 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.324420 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.324426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.324767 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.325556 (+ 789us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.326003 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":2184,"spinlock_wait_cycles":59520,"thread_start_us":146,"threads_started":1}
I20260620 04:36:01.327059 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.308140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.98.2:50709)
0620 04:36:01.310664 (+ 2524us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.310677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.311121 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.311843 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.311852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.312499 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.313300 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.313321 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.314615 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.314619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.316177 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.316208 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.317012 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.317015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.317232 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.318062 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.318087 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.321244 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.323347 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.323350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.323352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.323616 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.324043 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.324046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.324048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.324126 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.324614 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.324616 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.324762 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.325556 (+ 794us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.325956 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2442,"mutex_wait_us":1877,"spinlock_wait_cycles":33280,"thread_start_us":113,"threads_started":1}
I20260620 04:36:01.328477 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44907
I20260620 04:36:01.328698 7612 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.329135 7612 heartbeater.cc:511] Master 127.0.98.60:44907 requested a full tablet report, sending...
I20260620 04:36:01.329535 7613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:36:01.329592 7613 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.329702 7613 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:36:01.328477 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35623
I20260620 04:36:01.329890 7614 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.330165 7296 ts_manager.cc:194] Registered new tserver with Master: ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
I20260620 04:36:01.330241 6496 ts_manager.cc:194] Re-registered known tserver with Master: ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
I20260620 04:36:01.330843 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42927
I20260620 04:36:01.330029 7614 heartbeater.cc:511] Master 127.0.98.61:35623 requested a full tablet report, sending...
I20260620 04:36:01.331967 7105 ts_manager.cc:194] Registered new tserver with Master: ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
W20260620 04:36:01.431012 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:01.431270 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:01.431339 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:01.435317 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:01.435389 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:01.435496 7619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:01.440689 7619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35141
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:35623,127.0.98.60:44907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:01.441758 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:01.442696 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:01.452181 7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:01.452243 7619 server_base.cc:1034] running on GCE node
W20260620 04:36:01.452176 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:01.452176 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:01.452840 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:01.453610 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:01.454841 7619 hybrid_clock.cc:648] HybridClock initialized: now 1781930161454822 us; error 60 us; skew 500 ppm
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:01.457931 7619 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:36:01.458801 7619 webserver.cc:460] Webserver started at http://127.0.98.3:35141/ using document root <none> and password file <none>
I20260620 04:36:01.459317 7619 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:01.459391 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:01.463135 7619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:01.465143 7634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.466482 7619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:01.466631 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "2456e20bdc3f4822aa10812bf0dcc6c5"
format_stamp: "Formatted at 2026-06-20 04:36:00 on dist-test-slave-zmf4"
server_key: "d9a73a6a1266b747fb600b2c6c5775c6"
server_key_iv: "92e44f2bf91eee748c9103dc2ec38477"
server_key_version: "encryptionkey@0"
I20260620 04:36:01.467262 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:01.481096 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.477725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.98.60:44907)
0620 04:36:01.477854 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.477861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.477928 (+ 67us) server_negotiation.cc:407] Connection header received
0620 04:36:01.478096 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.478099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.478158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.478256 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.478499 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.479013 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.480008 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.480598 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.480711 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.480850 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.480899 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.480963 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:36:01.481113 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.477638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:59466)
0620 04:36:01.477846 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.477862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.477983 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.478267 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.478270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.478294 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.478402 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.478407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.479126 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.479129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.479890 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.479897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.480714 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.480718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.480804 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.480827 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.480890 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.480963 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":55,"threads_started":1}
I20260620 04:36:01.481915 7306 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 04:36:01.482158 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22458d73442e4504a2d0dac7c3be5c29. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:01.482290 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:36:01.488775 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:01.492097 7619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:01.492300 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:01.492852 7619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:01.493671 7619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:01.493738 7619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.493804 7619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:01.493849 7619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:01.502879 7619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34757
I20260620 04:36:01.502911 7749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34757 every 8 connection(s)
I20260620 04:36:01.503830 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:01.512298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7619
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:36:01.522502 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43907 (local address 127.0.98.61:35623)
0620 04:36:01.506998 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.507003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.507336 (+ 333us) server_negotiation.cc:407] Connection header received
0620 04:36:01.508698 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.508703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.508756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.508838 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.510015 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.510534 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512126 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.512495 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512570 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.518943 (+ 6373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.518947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.518949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.518976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.520657 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.521186 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.521191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.521195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.521282 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.521595 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.521598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.521600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.521810 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.521901 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.522233 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.522340 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260620 04:36:01.523141 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.98.3:43907)
0620 04:36:01.507180 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.507236 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.508204 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.508862 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.508867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.509300 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.509861 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.509868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.510670 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.510672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.511987 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.512006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.512752 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.512956 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.513692 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.513712 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.518850 (+ 5138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.520787 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.520790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.520792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.521061 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.521393 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.521396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.521400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.521485 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.521948 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.521953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.522066 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.522183 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.522310 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":170,"spinlock_wait_cycles":23168,"thread_start_us":119,"threads_started":1}
I20260620 04:36:01.523275 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36339 (local address 127.0.98.60:44907)
0620 04:36:01.506896 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.506903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.507807 (+ 904us) server_negotiation.cc:407] Connection header received
0620 04:36:01.508449 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.508453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.508507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.508572 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.509961 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.510855 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512900 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.513243 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.513333 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.516475 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.516481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.516486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.516519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.518255 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.518965 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.518968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.518970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.519061 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.519962 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.519967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.519969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.520241 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.520723 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.522482 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.522832 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442}
I20260620 04:36:01.523605 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.98.3:36339)
0620 04:36:01.507733 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.507746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.508249 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.508739 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.508747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.509294 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.509815 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.509828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.511939 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.511942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.512779 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.512786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.513521 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.513525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.513639 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.514108 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.514122 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.516278 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.518379 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.518384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.518399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.518751 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.519164 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.519167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.519169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.519599 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.520785 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.520792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.521132 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.521352 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.521702 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"mutex_wait_us":295,"spinlock_wait_cycles":1280,"thread_start_us":99,"threads_started":1}
Jun 20 04:36:01 dist-test-slave-zmf4 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930161, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:01.524742 7750 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44907
I20260620 04:36:01.524976 7750 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.525563 7750 heartbeater.cc:511] Master 127.0.98.60:44907 requested a full tablet report, sending...
I20260620 04:36:01.526175 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35623
I20260620 04:36:01.526276 7752 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.526394 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.514463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.98.61:35623)
0620 04:36:01.515847 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.515851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.515864 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:01.515897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.515903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.515944 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.516047 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.517015 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.517864 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.519273 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.519610 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.519707 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.521727 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.521731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.521734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.521757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.524098 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.524617 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.524621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.524623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.524700 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.525172 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.525176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.525179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.525375 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.525499 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.526105 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.526226 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":86,"threads_started":1}
I20260620 04:36:01.526463 7752 heartbeater.cc:511] Master 127.0.98.61:35623 requested a full tablet report, sending...
I20260620 04:36:01.526840 7296 ts_manager.cc:194] Registered new tserver with Master: 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3:34757)
I20260620 04:36:01.527503 7105 ts_manager.cc:194] Registered new tserver with Master: 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3:34757)
I20260620 04:36:01.529524 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:56787)
0620 04:36:01.507321 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.507333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.508204 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.508739 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.508746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.509300 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:01.509860 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.509868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.510970 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.510973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.512062 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.512072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.512939 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.513036 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.514063 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:01.514084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:01.525339 (+ 11255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:01.527614 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.527619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.527622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.527885 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.528227 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:01.528233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:01.528238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:01.528343 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:01.528852 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:01.528855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:01.528966 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.529180 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.529328 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":269,"spinlock_wait_cycles":55040,"thread_start_us":109,"threads_started":1}
I20260620 04:36:01.529572 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.506168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56787 (local address 127.0.98.62:36763)
0620 04:36:01.506302 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.506307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.507387 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 04:36:01.508416 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.508423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.508484 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.508572 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.510019 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.510858 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512250 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.512590 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.512681 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.525498 (+ 12817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.525503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.525507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.525548 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.527483 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.527988 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.527999 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.528000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.528089 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.528453 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.528457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.528459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.528695 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.528806 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.529250 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.529426 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:36:01.530205 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260620 04:36:01.530294 7751 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:01.530457 7751 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260620 04:36:01.530591 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.514678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.60:44907)
0620 04:36:01.516282 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.516296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.516308 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:36:01.516346 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.516348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.516392 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.516495 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.517308 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.518077 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.519342 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.519728 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.520298 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.521752 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.521755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.521757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.521779 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.528574 (+ 6795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.529152 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.529158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.529160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.529253 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.529602 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.529607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.529611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.529849 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.529974 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.530320 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.530438 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":95,"threads_started":1}
I20260620 04:36:01.531147 6496 ts_manager.cc:194] Re-registered known tserver with Master: 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3:34757)
I20260620 04:36:01.531853 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56787
I20260620 04:36:01.531904 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.515356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.98.62:36763)
0620 04:36:01.516698 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.516704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.516717 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:01.516753 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.516756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.516805 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.516900 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.517796 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.518549 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.524268 (+ 5719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.524595 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.524691 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.527392 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.527397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.527400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.527429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.529494 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.530696 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.530701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.530703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.530800 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.531078 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.531081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.531083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.531361 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.531488 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.531641 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.531753 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":72,"threads_started":1}
I20260620 04:36:01.538388 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:36:01.542274 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:01.544904 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: true } } }
I20260620 04:36:01.545075 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:01.548027 7769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: attempting to promote NON_VOTER 916d41cbd01542268848383252e2a3a8 to VOTER
W20260620 04:36:01.548527 7777 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "916d41cbd01542268848383252e2a3a8" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260620 04:36:01.549404 7769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } }
I20260620 04:36:01.550523 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:36:01.550627 7306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 LEARNER]: Refusing update from remote peer 22458d73442e4504a2d0dac7c3be5c29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:36:01.551062 7639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 04:36:01.551401 7639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 04:36:01.552948 7639 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 916d41cbd01542268848383252e2a3a8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.553488 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 916d41cbd01542268848383252e2a3a8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.553486 7306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 916d41cbd01542268848383252e2a3a8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.554435 7639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.554545 7639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:01.554692 7639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.554816 7639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:01.555132 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.555230 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:01.555984 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.556099 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:01.556155 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22458d73442e4504a2d0dac7c3be5c29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:01.556668 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:01.562865 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.558088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34511 (local address 127.0.0.1:58120)
0620 04:36:01.558246 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.558258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.558321 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.559093 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.559096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.559113 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.559210 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.559217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.560559 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.560562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.561429 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.561436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.562472 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.562475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.562558 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.562578 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.562650 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.562721 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 04:36:01.562878 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.558268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.2:34511)
0620 04:36:01.558604 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.558614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.558634 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:36:01.558730 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.558738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.558964 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.559143 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.559302 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.560440 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.561549 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.562359 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.562473 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.562622 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.562673 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.562754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
I20260620 04:36:01.563932 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.558695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39113 (local address 127.0.0.1:54594)
0620 04:36:01.559229 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.559240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.559311 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.559937 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.559940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.559957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.560039 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.560045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.561331 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.561334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.562480 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.562489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.563574 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.563578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.563677 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.563700 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.563772 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.563842 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":77,"threads_started":1}
I20260620 04:36:01.564368 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.558844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.98.1:39113)
0620 04:36:01.559532 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.559547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.559568 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:36:01.559638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.559643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.559811 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.559963 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.560125 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.561205 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.562623 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.563448 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.563544 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.563662 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.564141 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.564254 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":55,"threads_started":1}
I20260620 04:36:01.564574 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.558641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34757 (local address 127.0.0.1:48476)
0620 04:36:01.558940 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.558955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.559054 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.560707 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.560710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.560728 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.560801 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.560806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.562092 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.562096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.563173 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.563182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.564228 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.564232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.564316 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.564338 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.564386 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.564447 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20260620 04:36:01.565203 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.559033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.0.98.3:34757)
0620 04:36:01.560282 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.560293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.560313 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:36:01.560380 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.560385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.560583 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.560723 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.560889 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.561965 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.563320 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.564072 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.564823 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.564946 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.564992 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.565073 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":90,"threads_started":1}
I20260620 04:36:01.566190 7546 tablet_service.cc:1467] Processing CreateTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa (DEFAULT_TABLE table=test-table [id=738ebf6308d94c4dbb82ccf6cbf1f061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:01.567107 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e5ea4bb8f2481e875aa16a9488f7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:01.567960 7684 tablet_service.cc:1467] Processing CreateTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa (DEFAULT_TABLE table=test-table [id=738ebf6308d94c4dbb82ccf6cbf1f061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:01.568972 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e5ea4bb8f2481e875aa16a9488f7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:01.573348 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa (DEFAULT_TABLE table=test-table [id=738ebf6308d94c4dbb82ccf6cbf1f061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:01.574312 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e5ea4bb8f2481e875aa16a9488f7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:01.575178 7784 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Bootstrap starting.
I20260620 04:36:01.576423 7785 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Bootstrap starting.
I20260620 04:36:01.577513 7784 tablet_bootstrap.cc:654] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:01.578199 7784 log.cc:824] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:01.578444 7785 tablet_bootstrap.cc:654] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:01.579228 7785 log.cc:824] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:01.580317 7784 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: No bootstrap required, opened a new log
I20260620 04:36:01.580534 7784 ts_tablet_manager.cc:1397] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:36:01.581382 7788 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Bootstrap starting.
I20260620 04:36:01.582981 7788 tablet_bootstrap.cc:654] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:01.583595 7788 log.cc:824] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:01.584802 7784 raft_consensus.cc:348] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.585034 7784 raft_consensus.cc:374] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:01.585103 7784 raft_consensus.cc:729] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3df9fc4a504cbfad5a4316b6ce3d40, State: Initialized, Role: FOLLOWER
I20260620 04:36:01.585134 7788 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: No bootstrap required, opened a new log
I20260620 04:36:01.585318 7788 ts_tablet_manager.cc:1397] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 04:36:01.585592 7784 consensus_queue.cc:260] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.586638 7785 tablet_bootstrap.cc:492] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: No bootstrap required, opened a new log
I20260620 04:36:01.586643 7784 ts_tablet_manager.cc:1428] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:36:01.586807 7785 ts_tablet_manager.cc:1397] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260620 04:36:01.587779 7613 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260620 04:36:01.590533 7785 raft_consensus.cc:348] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.590787 7785 raft_consensus.cc:374] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:01.590850 7785 raft_consensus.cc:729] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2456e20bdc3f4822aa10812bf0dcc6c5, State: Initialized, Role: FOLLOWER
I20260620 04:36:01.591307 7785 consensus_queue.cc:260] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.592163 7785 ts_tablet_manager.cc:1428] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 04:36:01.592440 7751 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260620 04:36:01.593058 7788 raft_consensus.cc:348] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.593295 7788 raft_consensus.cc:374] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:01.593353 7788 raft_consensus.cc:729] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e46b13c7c9548dd9a168ca46978f838, State: Initialized, Role: FOLLOWER
I20260620 04:36:01.593770 7788 consensus_queue.cc:260] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.594674 7788 ts_tablet_manager.cc:1428] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 04:36:01.594947 7476 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
W20260620 04:36:01.599809 7478 tablet.cc:2369] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:01.621536 7791 raft_consensus.cc:482] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:01.621763 7791 raft_consensus.cc:504] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.622833 7791 leader_election.cc:288] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511), 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
I20260620 04:36:01.626566 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.623210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60183 (local address 127.0.98.2:34511)
0620 04:36:01.623379 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.623385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.623402 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:36:01.623543 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.623546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.623594 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.623672 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.623908 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.624486 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.625628 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.626020 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.626129 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.626197 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.626359 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.626431 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 04:36:01.626577 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.623205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38251 (local address 127.0.98.1:39113)
0620 04:36:01.623328 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.623332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.623347 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:01.623487 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.623491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.623537 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.623617 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:01.624224 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.624701 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.625661 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.626049 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.626167 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.626223 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.626384 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.626454 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:36:01.626626 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.623098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34511 (local address 127.0.98.3:60183)
0620 04:36:01.623338 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.623353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.623433 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.623687 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.623690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.623706 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.623810 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.623815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.624586 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.624592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.625500 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.625509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.626142 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.626145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.626242 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.626262 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.626345 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.626465 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":153}
I20260620 04:36:01.626628 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.623098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39113 (local address 127.0.98.3:38251)
0620 04:36:01.623254 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:01.623272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:01.623376 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:01.623635 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:01.623638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:01.623655 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:01.624096 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.624102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.624811 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.624814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.625525 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:01.625533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.626167 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.626170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:01.626272 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.626288 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:36:01.626370 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:36:01.626466 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":5}
I20260620 04:36:01.627262 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e5ea4bb8f2481e875aa16a9488f7aa" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" is_pre_election: true
I20260620 04:36:01.627424 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e5ea4bb8f2481e875aa16a9488f7aa" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e46b13c7c9548dd9a168ca46978f838" is_pre_election: true
I20260620 04:36:01.627570 7566 raft_consensus.cc:2454] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 0.
I20260620 04:36:01.627652 7429 raft_consensus.cc:2454] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 0.
I20260620 04:36:01.628052 7638 leader_election.cc:302] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2456e20bdc3f4822aa10812bf0dcc6c5, 3e46b13c7c9548dd9a168ca46978f838, ab3df9fc4a504cbfad5a4316b6ce3d40; no voters:
I20260620 04:36:01.628372 7791 raft_consensus.cc:2781] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:01.628469 7791 raft_consensus.cc:482] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:01.628530 7791 raft_consensus.cc:3037] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:01.629525 7791 raft_consensus.cc:504] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.629854 7791 leader_election.cc:288] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 election: Requested vote from peers ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511), 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113)
I20260620 04:36:01.630219 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e5ea4bb8f2481e875aa16a9488f7aa" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e46b13c7c9548dd9a168ca46978f838"
I20260620 04:36:01.630219 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e5ea4bb8f2481e875aa16a9488f7aa" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
I20260620 04:36:01.630350 7429 raft_consensus.cc:3037] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:01.630350 7566 raft_consensus.cc:3037] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:01.631564 7429 raft_consensus.cc:2454] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 1.
I20260620 04:36:01.631564 7566 raft_consensus.cc:2454] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 1.
I20260620 04:36:01.631899 7637 leader_election.cc:302] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2456e20bdc3f4822aa10812bf0dcc6c5, 3e46b13c7c9548dd9a168ca46978f838, ab3df9fc4a504cbfad5a4316b6ce3d40; no voters:
I20260620 04:36:01.632117 7791 raft_consensus.cc:2781] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:01.632357 7791 raft_consensus.cc:686] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 LEADER]: Becoming Leader. State: Replica: 2456e20bdc3f4822aa10812bf0dcc6c5, State: Running, Role: LEADER
I20260620 04:36:01.632710 7791 consensus_queue.cc:237] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:01.636168 6497 catalog_manager.cc:5495] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } health_report { overall_health: HEALTHY } } }
I20260620 04:36:01.661166 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.657561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.0.98.3:34757)
0620 04:36:01.657693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.657697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.657712 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:01.657785 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.657788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.657849 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.657961 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:01.658195 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.658700 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.659584 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.659902 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.660041 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.660376 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:01.660753 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:01.660887 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.660964 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.661018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:36:01.666879 7566 raft_consensus.cc:1261] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Refusing update from remote peer 2456e20bdc3f4822aa10812bf0dcc6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:01.666939 7429 raft_consensus.cc:1261] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Refusing update from remote peer 2456e20bdc3f4822aa10812bf0dcc6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:01.667614 7793 consensus_queue.cc:1035] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:01.667852 7791 consensus_queue.cc:1035] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:01.676043 7796 mvcc.cc:205] Tried to move back new op lower bound from 7298785942180233216 to 7298785942051016704. Current Snapshot: MvccSnapshot[applied={T|T < 7298785942180233216}]
I20260620 04:36:01.677120 7798 mvcc.cc:205] Tried to move back new op lower bound from 7298785942180233216 to 7298785942051016704. Current Snapshot: MvccSnapshot[applied={T|T < 7298785942180233216}]
I20260620 04:36:01.679780 7797 mvcc.cc:205] Tried to move back new op lower bound from 7298785942180233216 to 7298785942051016704. Current Snapshot: MvccSnapshot[applied={T|T < 7298785942180233216}]
I20260620 04:36:01.684914 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:01.685075 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:01.690318 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22458d73442e4504a2d0dac7c3be5c29: Sending DeleteTablet for 3 replicas of tablet 91e5ea4bb8f2481e875aa16a9488f7aa
I20260620 04:36:01.691027 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54594
I20260620 04:36:01.691190 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48476
I20260620 04:36:01.691190 7546 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e5ea4bb8f2481e875aa16a9488f7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58120
I20260620 04:36:01.691637 7807 tablet_replica.cc:330] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: stopping tablet replica
I20260620 04:36:01.691959 7807 raft_consensus.cc:2229] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:01.692472 7807 raft_consensus.cc:2258] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:01.693248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6465
I20260620 04:36:01.694466 7807 ts_tablet_manager.cc:1905] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:01.696411 7806 tablet_replica.cc:330] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: stopping tablet replica
I20260620 04:36:01.696800 7806 raft_consensus.cc:2229] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:01.697149 7807 ts_tablet_manager.cc:1918] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:01.697245 7807 log.cc:1196] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/91e5ea4bb8f2481e875aa16a9488f7aa
I20260620 04:36:01.697369 7806 raft_consensus.cc:2258] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:01.697556 7807 ts_tablet_manager.cc:1939] T 91e5ea4bb8f2481e875aa16a9488f7aa P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting consensus metadata
I20260620 04:36:01.698897 7806 ts_tablet_manager.cc:1905] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:36:01.701086 7499 connection.cc:535] server connection from 127.0.0.1:58120 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:36:01.701550 7806 ts_tablet_manager.cc:1918] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:01.701646 7806 log.cc:1196] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/91e5ea4bb8f2481e875aa16a9488f7aa
I20260620 04:36:01.701977 7806 ts_tablet_manager.cc:1939] T 91e5ea4bb8f2481e875aa16a9488f7aa P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting consensus metadata
W20260620 04:36:01.703061 7636 connection.cc:441] server connection from 127.0.0.1:48476 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48476 (request call id 1) could send its response
I20260620 04:36:01.703271 7805 tablet_replica.cc:330] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: stopping tablet replica
I20260620 04:36:01.703605 7805 raft_consensus.cc:2229] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:01.703958 7805 raft_consensus.cc:2258] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:01.705605 7805 ts_tablet_manager.cc:1905] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:01.708441 7805 ts_tablet_manager.cc:1918] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:01.708544 7805 log.cc:1196] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/91e5ea4bb8f2481e875aa16a9488f7aa
I20260620 04:36:01.708876 7805 ts_tablet_manager.cc:1939] T 91e5ea4bb8f2481e875aa16a9488f7aa P 3e46b13c7c9548dd9a168ca46978f838: Deleting consensus metadata
W20260620 04:36:01.709944 7364 connection.cc:441] server connection from 127.0.0.1:54594 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54594 (request call id 1) could send its response
I20260620 04:36:01.713053 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.692781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.0.98.61:35623)
0620 04:36:01.692893 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.692898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.697938 (+ 5040us) server_negotiation.cc:407] Connection header received
0620 04:36:01.698147 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.698153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.698217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.698274 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.699402 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.700261 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.702031 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.702367 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.702447 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.705372 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.705376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.705380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.705406 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.707246 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.707730 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.707733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.707736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.707817 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.708084 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.708087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.708089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.708324 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.708439 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.712747 (+ 4308us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.712873 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:36:01.717372 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.704505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.98.60:44907)
0620 04:36:01.704627 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.704631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.706262 (+ 1631us) server_negotiation.cc:407] Connection header received
0620 04:36:01.706428 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.706431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.706478 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.706527 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.708641 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.709400 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.710752 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.711080 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.711150 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.712841 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.712845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.712848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.712871 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.714880 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.715941 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.715947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.715949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.716037 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.716375 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.716380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.716384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.716590 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.716705 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.717080 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.717196 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:36:01.718297 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.705613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.0.98.61:35623)
0620 04:36:01.705720 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.705724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.707540 (+ 1816us) server_negotiation.cc:407] Connection header received
0620 04:36:01.707708 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.707712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.707751 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.707798 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.710682 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.711394 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.712666 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.712984 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.713075 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.714880 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.714884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.714887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.714911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.716556 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.717025 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.717028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.717030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.717106 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.717406 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.717411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.717413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.717608 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.717707 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.718013 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.718117 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:36:01.721921 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:01.692881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.0.98.60:44907)
0620 04:36:01.692987 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:01.692991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:01.710401 (+ 17410us) server_negotiation.cc:407] Connection header received
0620 04:36:01.710441 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:01.710444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:01.710492 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:01.710569 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:01.711412 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.712555 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.715247 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:01.715594 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:01.715688 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:01.717250 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:01.717254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:01.717257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:01.717281 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:01.720138 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.720659 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.720662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.720663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.720754 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:01.721032 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:01.721034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:01.721035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:01.721206 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:01.721306 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:01.721611 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 04:36:01.721755 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:36:02.682137 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:02.681636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:42103)
0620 04:36:02.681924 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:02.682022 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
W20260620 04:36:02.682662 7751 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:02.683359 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:02.682750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:51413)
0620 04:36:02.683217 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:02.683285 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":71,"threads_started":1}
I20260620 04:36:02.683745 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:02.683375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:36311)
0620 04:36:02.683612 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:02.683673 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
W20260620 04:36:02.683753 7613 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
W20260620 04:36:02.684013 7476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.190083 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22458d73442e4504a2d0dac7c3be5c29)
I20260620 04:36:03.190253 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } }
I20260620 04:36:03.190303 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22458d73442e4504a2d0dac7c3be5c29)
I20260620 04:36:03.190512 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } }
I20260620 04:36:03.190820 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763), 6580f9d393b84455bff15c6e36e41894 (127.0.98.61:35623)
I20260620 04:36:03.191506 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.190999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60704)
0620 04:36:03.191313 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.191404 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":72,"threads_started":1}
W20260620 04:36:03.191709 7283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.192435 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763), 916d41cbd01542268848383252e2a3a8 (127.0.98.60:44907)
I20260620 04:36:03.192762 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.192452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60732)
0620 04:36:03.192594 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.192691 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 04:36:03.192860 7283 leader_election.cc:334] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.192881 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.191740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60720)
0620 04:36:03.192723 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.192791 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":68,"threads_started":1}
W20260620 04:36:03.193039 7092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.193956 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.193722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60742)
0620 04:36:03.193851 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.193898 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 04:36:03.194065 7092 leader_election.cc:334] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.196378 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.191036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35623 (local address 127.0.0.1:39980)
0620 04:36:03.191357 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.191372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:03.191512 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:03.191902 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:03.191904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:03.191921 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:03.192030 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.192037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.193342 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.193348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.194473 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.194509 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.195681 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.195688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.195826 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.195879 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 04:36:03.196137 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 04:36:03.196255 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260620 04:36:03.196396 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.191088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.0.98.61:35623)
0620 04:36:03.191339 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.191344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.191458 (+ 114us) server_negotiation.cc:407] Connection header received
0620 04:36:03.191740 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.191743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.191805 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.191893 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:03.192161 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.193243 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.194704 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.195467 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.195564 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.195611 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.196174 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.196303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260620 04:36:03.196758 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "916d41cbd01542268848383252e2a3a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6580f9d393b84455bff15c6e36e41894" is_pre_election: true
I20260620 04:36:03.196905 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916d41cbd01542268848383252e2a3a8 in term 1.
I20260620 04:36:03.197184 7282 leader_election.cc:302] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6580f9d393b84455bff15c6e36e41894, 916d41cbd01542268848383252e2a3a8; no voters: 22458d73442e4504a2d0dac7c3be5c29
I20260620 04:36:03.197366 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 04:36:03.197446 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22458d73442e4504a2d0dac7c3be5c29)
I20260620 04:36:03.197623 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:36:03.198098 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.192265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:59512)
0620 04:36:03.192642 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.192657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:03.192753 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:03.193576 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:03.193579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:03.193598 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:03.193677 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.193682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.194536 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.194543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.196433 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.196449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.197630 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.197637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.197759 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.197789 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 04:36:03.197889 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 04:36:03.197973 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":96,"thread_start_us":61,"threads_started":1}
I20260620 04:36:03.198108 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.192382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.60:44907)
0620 04:36:03.192628 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.192636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.192846 (+ 210us) server_negotiation.cc:407] Connection header received
0620 04:36:03.192891 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.192894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.192940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.193576 (+ 636us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:03.193790 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.194431 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.196673 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.197279 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.197408 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.197551 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.197927 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.198014 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260620 04:36:03.198894 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } }
I20260620 04:36:03.199270 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 election: Requested vote from peers 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763), 6580f9d393b84455bff15c6e36e41894 (127.0.98.61:35623)
I20260620 04:36:03.199673 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "916d41cbd01542268848383252e2a3a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6580f9d393b84455bff15c6e36e41894"
I20260620 04:36:03.199692 7306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6580f9d393b84455bff15c6e36e41894" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "916d41cbd01542268848383252e2a3a8" is_pre_election: true
I20260620 04:36:03.199797 7115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:36:03.199898 7306 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6580f9d393b84455bff15c6e36e41894 in current term 2: Already voted for candidate 916d41cbd01542268848383252e2a3a8 in this term.
I20260620 04:36:03.200106 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.199347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60746)
0620 04:36:03.199479 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.199532 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 04:36:03.200325 7090 leader_election.cc:302] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6580f9d393b84455bff15c6e36e41894; no voters: 22458d73442e4504a2d0dac7c3be5c29, 916d41cbd01542268848383252e2a3a8
I20260620 04:36:03.200752 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.200539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60750)
0620 04:36:03.200645 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.200697 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 04:36:03.200846 7283 leader_election.cc:334] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260620 04:36:03.201048 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916d41cbd01542268848383252e2a3a8 in term 2.
I20260620 04:36:03.201171 7820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:36:03.201311 7282 leader_election.cc:302] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6580f9d393b84455bff15c6e36e41894, 916d41cbd01542268848383252e2a3a8; no voters: 22458d73442e4504a2d0dac7c3be5c29
I20260620 04:36:03.201486 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:36:03.201679 7819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [term 2 LEADER]: Becoming Leader. State: Replica: 916d41cbd01542268848383252e2a3a8, State: Running, Role: LEADER
I20260620 04:36:03.201911 7819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } }
I20260620 04:36:03.202921 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916d41cbd01542268848383252e2a3a8. Latest consensus state: current_term: 2 leader_uuid: "916d41cbd01542268848383252e2a3a8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:03.203056 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:03.203351 7831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:03.205487 7831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:03.205870 7831 catalog_manager.cc:1259] Loaded cluster ID: 9f419d9d39334c208c7d601f57d82828
I20260620 04:36:03.205917 7831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:03.206432 7831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:03.206831 7831 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Loaded TSK: 0
I20260620 04:36:03.207466 7831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:36:03.228178 7295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:36:03.231220 7295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:03.234133 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.233825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60762)
0620 04:36:03.233986 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.234054 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:36:03.234148 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [term 2 FOLLOWER]: Refusing update from remote peer 916d41cbd01542268848383252e2a3a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 04:36:03.234602 7819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 04:36:03.234916 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.234608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:60772)
0620 04:36:03.234731 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.234779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 04:36:03.235303 7283 consensus_peers.cc:487] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 -> Peer 22458d73442e4504a2d0dac7c3be5c29 (127.0.98.62:36763): Couldn't send request to peer 22458d73442e4504a2d0dac7c3be5c29. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 04:36:03.236497 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916d41cbd01542268848383252e2a3a8. Latest consensus state: current_term: 2 leader_uuid: "916d41cbd01542268848383252e2a3a8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:03.236650 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:03.237711 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "916d41cbd01542268848383252e2a3a8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:03.237829 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:03.238006 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "916d41cbd01542268848383252e2a3a8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "22458d73442e4504a2d0dac7c3be5c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "6580f9d393b84455bff15c6e36e41894" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35623 } attrs { promote: false } } peers { permanent_uuid: "916d41cbd01542268848383252e2a3a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attrs { promote: false } } }
I20260620 04:36:03.238121 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6580f9d393b84455bff15c6e36e41894 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:36:03.247757 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.243210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39113 (local address 127.0.0.1:54602)
0620 04:36:03.243316 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.243335 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:03.243398 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:03.243871 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:03.243876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:03.243896 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:03.244026 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.244032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.245301 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.245305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.246502 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.246510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.247376 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.247380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.247476 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.247501 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:36:03.247566 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:36:03.247641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:36:03.247938 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.243816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34757 (local address 127.0.0.1:48486)
0620 04:36:03.244067 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.244081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:03.244151 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:03.244934 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:03.244940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:03.244957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:03.245039 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.245044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.245884 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.245889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.246662 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.246671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.247512 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.247516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.247648 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.247672 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:36:03.247740 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:36:03.247808 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260620 04:36:03.248004 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.244415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.98.3:34757)
0620 04:36:03.244692 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.244695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.244706 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:36:03.244747 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.244748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.244809 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.245123 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:03.245233 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.245784 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.246797 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.247407 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.247498 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.247579 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.247770 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.247853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260620 04:36:03.248384 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.243342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.98.1:39113)
0620 04:36:03.243611 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.243616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.243631 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:03.243684 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.243687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.243748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.244362 (+ 614us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:03.244472 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.245183 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.246649 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.247261 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.247367 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.247428 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.248132 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.248253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260620 04:36:03.248582 7684 tablet_service.cc:1467] Processing CreateTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d (DEFAULT_TABLE table=test-table [id=f1acc009096e49a7b2e983508827c825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:03.248848 7409 tablet_service.cc:1467] Processing CreateTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d (DEFAULT_TABLE table=test-table [id=f1acc009096e49a7b2e983508827c825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:03.248976 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69d677dee3d4f3aa86d7140ee5ede1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:03.249121 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69d677dee3d4f3aa86d7140ee5ede1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:03.249280 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.243614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34511 (local address 127.0.0.1:58122)
0620 04:36:03.244035 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.244051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:03.244137 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:03.245239 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:03.245242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:03.245256 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:03.245332 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.245337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.246389 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.246393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.247550 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:03.247558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.248893 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.248899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:03.249016 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.249039 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:36:03.249110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:36:03.249169 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":28}
I20260620 04:36:03.249689 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.243752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.0.98.2:34511)
0620 04:36:03.245023 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.245027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.245041 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:03.245089 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.245092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.245133 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.245225 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:03.245477 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.246260 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.247674 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.248576 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.248767 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.248830 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.249480 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.249560 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":69,"threads_started":1}
I20260620 04:36:03.250149 7546 tablet_service.cc:1467] Processing CreateTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d (DEFAULT_TABLE table=test-table [id=f1acc009096e49a7b2e983508827c825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:03.250399 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69d677dee3d4f3aa86d7140ee5ede1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:03.251662 7842 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838: Bootstrap starting.
I20260620 04:36:03.252103 7843 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Bootstrap starting.
I20260620 04:36:03.252532 7842 tablet_bootstrap.cc:654] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:03.253046 7843 tablet_bootstrap.cc:654] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:03.253479 7845 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Bootstrap starting.
I20260620 04:36:03.253533 7842 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838: No bootstrap required, opened a new log
I20260620 04:36:03.253814 7842 ts_tablet_manager.cc:1397] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:03.254199 7843 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: No bootstrap required, opened a new log
I20260620 04:36:03.254287 7843 ts_tablet_manager.cc:1397] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:03.254357 7845 tablet_bootstrap.cc:654] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:03.254542 7842 raft_consensus.cc:348] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.254654 7842 raft_consensus.cc:374] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:03.254693 7842 raft_consensus.cc:729] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e46b13c7c9548dd9a168ca46978f838, State: Initialized, Role: FOLLOWER
I20260620 04:36:03.254733 7843 raft_consensus.cc:348] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.254863 7843 raft_consensus.cc:374] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:03.254824 7842 consensus_queue.cc:260] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.254914 7843 raft_consensus.cc:729] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2456e20bdc3f4822aa10812bf0dcc6c5, State: Initialized, Role: FOLLOWER
I20260620 04:36:03.255046 7843 consensus_queue.cc:260] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.255214 7842 ts_tablet_manager.cc:1428] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:03.255402 7843 ts_tablet_manager.cc:1428] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:03.255456 7845 tablet_bootstrap.cc:492] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: No bootstrap required, opened a new log
I20260620 04:36:03.255764 7845 ts_tablet_manager.cc:1397] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:36:03.256261 7845 raft_consensus.cc:348] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.256371 7845 raft_consensus.cc:374] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:03.256407 7845 raft_consensus.cc:729] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3df9fc4a504cbfad5a4316b6ce3d40, State: Initialized, Role: FOLLOWER
W20260620 04:36:03.256492 7753 tablet.cc:2369] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:03.256543 7845 consensus_queue.cc:260] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.256623 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.255954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:57711)
0620 04:36:03.256362 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.256511 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":99,"threads_started":1}
I20260620 04:36:03.256886 7845 ts_tablet_manager.cc:1428] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 04:36:03.257440 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.257073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:56781)
0620 04:36:03.257298 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.257354 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260620 04:36:03.258455 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.258026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:56657)
0620 04:36:03.258286 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.258344 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
I20260620 04:36:03.259766 7750 heartbeater.cc:503] Master 127.0.98.60:44907 was elected leader, sending a full tablet report...
I20260620 04:36:03.260434 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.260030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:38525)
0620 04:36:03.260316 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.260366 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260620 04:36:03.261217 7612 heartbeater.cc:503] Master 127.0.98.60:44907 was elected leader, sending a full tablet report...
I20260620 04:36:03.261727 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.257294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:53695)
0620 04:36:03.261562 (+ 4268us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.261626 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4191,"thread_start_us":85,"threads_started":1}
I20260620 04:36:03.264602 7849 raft_consensus.cc:482] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:03.264710 7849 raft_consensus.cc:504] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.265089 7849 leader_election.cc:288] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113), ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
I20260620 04:36:03.265413 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d677dee3d4f3aa86d7140ee5ede1d" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" is_pre_election: true
I20260620 04:36:03.265573 7566 raft_consensus.cc:2454] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 0.
I20260620 04:36:03.265540 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d677dee3d4f3aa86d7140ee5ede1d" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e46b13c7c9548dd9a168ca46978f838" is_pre_election: true
I20260620 04:36:03.265663 7429 raft_consensus.cc:2454] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 0.
I20260620 04:36:03.265942 7638 leader_election.cc:302] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2456e20bdc3f4822aa10812bf0dcc6c5, 3e46b13c7c9548dd9a168ca46978f838; no voters:
I20260620 04:36:03.266129 7849 raft_consensus.cc:2781] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:03.266216 7849 raft_consensus.cc:482] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:03.266270 7849 raft_consensus.cc:3037] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:03.267202 7849 raft_consensus.cc:504] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.267500 7849 leader_election.cc:288] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113), ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511)
I20260620 04:36:03.267820 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d677dee3d4f3aa86d7140ee5ede1d" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e46b13c7c9548dd9a168ca46978f838"
I20260620 04:36:03.267817 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d677dee3d4f3aa86d7140ee5ede1d" candidate_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40"
I20260620 04:36:03.267940 7429 raft_consensus.cc:3037] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:03.267956 7566 raft_consensus.cc:3037] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:03.268823 7429 raft_consensus.cc:2454] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 1.
I20260620 04:36:03.268860 7566 raft_consensus.cc:2454] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2456e20bdc3f4822aa10812bf0dcc6c5 in term 1.
I20260620 04:36:03.269161 7637 leader_election.cc:302] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2456e20bdc3f4822aa10812bf0dcc6c5, ab3df9fc4a504cbfad5a4316b6ce3d40; no voters:
I20260620 04:36:03.269694 7849 raft_consensus.cc:2781] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:03.269869 7849 raft_consensus.cc:686] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 LEADER]: Becoming Leader. State: Replica: 2456e20bdc3f4822aa10812bf0dcc6c5, State: Running, Role: LEADER
I20260620 04:36:03.270022 7849 consensus_queue.cc:237] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } }
I20260620 04:36:03.270833 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.270566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:35103)
0620 04:36:03.270695 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.270750 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 04:36:03.271708 7296 catalog_manager.cc:5495] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2456e20bdc3f4822aa10812bf0dcc6c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } health_report { overall_health: HEALTHY } } }
I20260620 04:36:03.287253 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.284192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.98.3:34757)
0620 04:36:03.284324 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:03.284328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:03.284340 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:36:03.284455 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:03.284457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:03.284498 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:03.284560 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:03.284852 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.285458 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.286323 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:03.286532 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:03.286626 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:03.286840 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:03.286902 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:03.286954 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:03.287025 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:36:03.287068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:36:03.289304 7429 raft_consensus.cc:1261] T f69d677dee3d4f3aa86d7140ee5ede1d P 3e46b13c7c9548dd9a168ca46978f838 [term 1 FOLLOWER]: Refusing update from remote peer 2456e20bdc3f4822aa10812bf0dcc6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:03.289345 7566 raft_consensus.cc:1261] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Refusing update from remote peer 2456e20bdc3f4822aa10812bf0dcc6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:03.289814 7855 consensus_queue.cc:1035] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e46b13c7c9548dd9a168ca46978f838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:03.290025 7849 consensus_queue.cc:1035] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3df9fc4a504cbfad5a4316b6ce3d40" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:03.291219 7475 heartbeater.cc:503] Master 127.0.98.60:44907 was elected leader, sending a full tablet report...
I20260620 04:36:03.293777 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.293442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:41657)
0620 04:36:03.293636 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.293695 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 04:36:03.294497 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.291083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:36075)
0620 04:36:03.292836 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.292899 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1682}
I20260620 04:36:03.295516 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.295256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:57657)
0620 04:36:03.295378 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.295435 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:36:03.297408 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.297144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:48655)
0620 04:36:03.297289 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.297342 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 04:36:03.298661 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.298402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:34751)
0620 04:36:03.298534 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.298589 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 04:36:03.299243 7294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:03.299401 7294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:03.300225 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.299950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:37881)
0620 04:36:03.300081 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.300131 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 04:36:03.303335 7294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 916d41cbd01542268848383252e2a3a8: Sending DeleteTablet for 3 replicas of tablet f69d677dee3d4f3aa86d7140ee5ede1d
I20260620 04:36:03.303907 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54602
I20260620 04:36:03.304139 7546 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58122
I20260620 04:36:03.304255 7545 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58122
I20260620 04:36:03.304270 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d677dee3d4f3aa86d7140ee5ede1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48486
I20260620 04:36:03.304466 7868 tablet_replica.cc:330] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: stopping tablet replica
I20260620 04:36:03.304540 7869 tablet_replica.cc:330] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: stopping tablet replica
I20260620 04:36:03.304569 7868 raft_consensus.cc:2229] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:03.304662 7869 raft_consensus.cc:2229] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:03.304711 7868 raft_consensus.cc:2258] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:03.304840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7250
I20260620 04:36:03.306421 7868 ts_tablet_manager.cc:1905] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:03.308241 7869 raft_consensus.cc:2258] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:03.309756 7868 ts_tablet_manager.cc:1918] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:03.309855 7868 log.cc:1196] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/f69d677dee3d4f3aa86d7140ee5ede1d
I20260620 04:36:03.310128 7868 ts_tablet_manager.cc:1939] T f69d677dee3d4f3aa86d7140ee5ede1d P ab3df9fc4a504cbfad5a4316b6ce3d40: Deleting consensus metadata
I20260620 04:36:03.310796 7282 catalog_manager.cc:4841] TS ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511): tablet f69d677dee3d4f3aa86d7140ee5ede1d (table test-table [id=f1acc009096e49a7b2e983508827c825]) successfully deleted
W20260620 04:36:03.311177 7282 catalog_manager.cc:4808] TS ab3df9fc4a504cbfad5a4316b6ce3d40 (127.0.98.2:34511): delete failed for tablet f69d677dee3d4f3aa86d7140ee5ede1d because the tablet was not found. No further retry: Not found: Tablet not found: f69d677dee3d4f3aa86d7140ee5ede1d
I20260620 04:36:03.312666 7869 ts_tablet_manager.cc:1905] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:03.314661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7483
I20260620 04:36:03.314865 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:03.314549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39113 (local address 127.0.0.1:54614)
0620 04:36:03.314727 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:03.314779 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 04:36:03.315444 7283 catalog_manager.cc:4584] TS 3e46b13c7c9548dd9a168ca46978f838 (127.0.98.1:39113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f69d677dee3d4f3aa86d7140ee5ede1d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39113: connect: Connection refused (error 111)
I20260620 04:36:03.315843 7869 ts_tablet_manager.cc:1918] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:03.315896 7869 log.cc:1196] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/f69d677dee3d4f3aa86d7140ee5ede1d
I20260620 04:36:03.316349 7869 ts_tablet_manager.cc:1939] T f69d677dee3d4f3aa86d7140ee5ede1d P 2456e20bdc3f4822aa10812bf0dcc6c5: Deleting consensus metadata
I20260620 04:36:03.317204 7282 catalog_manager.cc:4841] TS 2456e20bdc3f4822aa10812bf0dcc6c5 (127.0.98.3:34757): tablet f69d677dee3d4f3aa86d7140ee5ede1d (table test-table [id=f1acc009096e49a7b2e983508827c825]) successfully deleted
I20260620 04:36:03.321198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7619
I20260620 04:36:03.328121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 6961
I20260620 04:36:03.333904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7069
2026-06-20T04:36:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5155 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:36:03 dist-test-slave-zmf4 krb5kdc[7872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:36:03 dist-test-slave-zmf4 krb5kdc[7872](info): set up 2 sockets
Jun 20 04:36:03 dist-test-slave-zmf4 krb5kdc[7872](info): commencing operation
krb5kdc: starting...
W20260620 04:36:04.404856 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:36:04 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:36:04.529170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:04.640089 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:04.640419 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:04.640463 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:04.644512 7885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:04.644565 7885 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:36:04.644582 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:04.644600 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:04.644616 7885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:04.644657 7885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:04.649858 7885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33601
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:04.650867 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:04.651835 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:04.660099 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:04.660221 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:04.660269 7885 server_base.cc:1034] running on GCE node
W20260620 04:36:04.660414 7893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:04.660950 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:04.661962 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:04.663132 7885 hybrid_clock.cc:648] HybridClock initialized: now 1781930164663113 us; error 31 us; skew 500 ppm
Jun 20 04:36:04 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:04.665959 7885 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:36:04.667025 7885 webserver.cc:460] Webserver started at http://127.0.98.62:39339/ using document root <none> and password file <none>
I20260620 04:36:04.667593 7885 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:04.667661 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:04.667876 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:04.670010 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "195e0f7463e54e7cb8169459bd7372db"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "1f3685e30ac945ee2c2d55be5956895c"
server_key_iv: "e3b80555996eab73403ea49b4b1f37dd"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.670595 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "195e0f7463e54e7cb8169459bd7372db"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "1f3685e30ac945ee2c2d55be5956895c"
server_key_iv: "e3b80555996eab73403ea49b4b1f37dd"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.674448 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:36:04.676764 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:04.678053 7885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:04.678282 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "195e0f7463e54e7cb8169459bd7372db"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "1f3685e30ac945ee2c2d55be5956895c"
server_key_iv: "e3b80555996eab73403ea49b4b1f37dd"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.678383 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:04.696609 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:04.699664 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:04.699862 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:04.707598 7885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33601
I20260620 04:36:04.707615 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33601 every 8 connection(s)
I20260620 04:36:04.708568 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:04.711369 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:04.715257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7885
I20260620 04:36:04.715359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:36:04.715660 392 external_mini_cluster.cc:1428] Setting key 351cafc920e36fc406077f94737ca376
I20260620 04:36:04.717060 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Bootstrap starting.
I20260620 04:36:04.719633 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:04.720402 7963 log.cc:824] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:04.722388 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: No bootstrap required, opened a new log
Jun 20 04:36:04 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:04.726106 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } }
I20260620 04:36:04.726356 7963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:04.726438 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195e0f7463e54e7cb8169459bd7372db, State: Initialized, Role: FOLLOWER
I20260620 04:36:04.726893 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } }
I20260620 04:36:04.727037 7963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:04.727116 7963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:04.727219 7963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:04.728056 7963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } }
I20260620 04:36:04.728420 7963 leader_election.cc:302] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195e0f7463e54e7cb8169459bd7372db; no voters:
I20260620 04:36:04.728771 7963 leader_election.cc:288] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:36:04.728904 7968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:04.729128 7968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [term 1 LEADER]: Becoming Leader. State: Replica: 195e0f7463e54e7cb8169459bd7372db, State: Running, Role: LEADER
I20260620 04:36:04.729453 7968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } }
I20260620 04:36:04.730023 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:04.717046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:33601)
0620 04:36:04.717462 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:04.717483 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:04.717515 (+ 32us) server_negotiation.cc:407] Connection header received
0620 04:36:04.718125 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:04.718140 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:04.718437 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:04.718708 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:04.719587 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.720675 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.721756 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.722229 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.722357 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:04.725136 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:04.725141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:04.725153 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:04.725177 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:04.727051 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.727550 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.727555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.727566 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.727668 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.727940 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.727945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.727947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.728407 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:04.728604 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:04.728866 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 04:36:04.729195 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260620 04:36:04.730053 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:04.730647 7968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195e0f7463e54e7cb8169459bd7372db. Latest consensus state: current_term: 1 leader_uuid: "195e0f7463e54e7cb8169459bd7372db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } } }
I20260620 04:36:04.730791 7968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:04.731140 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195e0f7463e54e7cb8169459bd7372db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e0f7463e54e7cb8169459bd7372db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33601 } } }
I20260620 04:36:04.731303 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db [sys.catalog]: This master's current role is: LEADER
W20260620 04:36:04.734802 7983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:36:04.734886 7983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:36:04.734972 7975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:04.735834 7975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:04.741673 7975 catalog_manager.cc:1347] Generated new cluster ID: 447987a816c547fa8213b641b59fb00e
I20260620 04:36:04.741745 7975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:04.755251 7975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:36:04.756551 7975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:04.768807 7975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Generated new TSK 0
I20260620 04:36:04.769747 7975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:36:04.795284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33601
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:04.905125 7989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:04.905406 7989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:04.905465 7989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:04.910181 7989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:04.910269 7989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:04.910313 7989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:36:04.910347 7989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:04.915555 7989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:04.916635 7989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:04.917598 7989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:04.927037 7994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:04.927062 7989 server_base.cc:1034] running on GCE node
W20260620 04:36:04.927016 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:04.927023 7995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:04.927608 7989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:04.928263 7989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:04.929451 7989 hybrid_clock.cc:648] HybridClock initialized: now 1781930164929402 us; error 61 us; skew 500 ppm
Jun 20 04:36:04 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:04.932361 7989 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:36:04.933174 7989 webserver.cc:460] Webserver started at http://127.0.98.1:44255/ using document root <none> and password file <none>
I20260620 04:36:04.933710 7989 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:04.933796 7989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:04.934015 7989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:04.935935 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "16704c72df034068816e50e2ea2bf41f"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "da7792d5abd9e7a5cc46c85578355a81"
server_key_iv: "4fdeb35bea079e1c3904e29177f91f10"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.936506 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16704c72df034068816e50e2ea2bf41f"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "da7792d5abd9e7a5cc46c85578355a81"
server_key_iv: "4fdeb35bea079e1c3904e29177f91f10"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.939966 7989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:04.942384 8004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:04.943658 7989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:04.943794 7989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "16704c72df034068816e50e2ea2bf41f"
format_stamp: "Formatted at 2026-06-20 04:36:04 on dist-test-slave-zmf4"
server_key: "da7792d5abd9e7a5cc46c85578355a81"
server_key_iv: "4fdeb35bea079e1c3904e29177f91f10"
server_key_version: "encryptionkey@0"
I20260620 04:36:04.943902 7989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:04.957849 7989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:04.961009 7989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:04.961191 7989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:04.962615 7989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:04.962682 7989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:04.962746 7989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:04.962792 7989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:36:04 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:04.973785 7989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43585
I20260620 04:36:04.974251 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43585 every 8 connection(s)
I20260620 04:36:04.974879 7989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:04.977337 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:04.963418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56649 (local address 127.0.98.62:33601)
0620 04:36:04.963541 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:04.963544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:04.964595 (+ 1051us) server_negotiation.cc:407] Connection header received
0620 04:36:04.965657 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:04.965675 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:04.965740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:04.965796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:04.967194 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.967856 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.969033 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.969292 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.969453 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:04.973135 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:04.973142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:04.973148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:04.973181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:04.975063 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.975690 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.975693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.975694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.975769 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.976147 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.976150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.976151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.976387 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:04.976472 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:04.977035 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 04:36:04.977138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:36:04.978143 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:04.963889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.1:56649)
0620 04:36:04.964450 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:04.964492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:04.965397 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:04.966100 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:04.966105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:04.966407 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:04.967050 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:04.967067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.967986 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.967990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:04.968909 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:04.968918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.969498 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.969505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:04.969721 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:04.970402 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:04.970430 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:04.972965 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:04.975173 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:04.975178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:04.975188 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:04.975553 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:04.975870 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:04.975872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:04.975874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:04.976013 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:04.976513 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:04.976519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:04.976761 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:36:04.976919 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:36:04.977181 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":148,"threads_started":1}
I20260620 04:36:04.979491 8120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33601
I20260620 04:36:04.979627 8120 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:04.980101 8120 heartbeater.cc:511] Master 127.0.98.62:33601 requested a full tablet report, sending...
I20260620 04:36:04.981446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 7989
I20260620 04:36:04.981580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:36:04.981684 7916 ts_manager.cc:194] Registered new tserver with Master: 16704c72df034068816e50e2ea2bf41f (127.0.98.1:43585)
I20260620 04:36:04.981892 392 external_mini_cluster.cc:1428] Setting key f05db8ff81f3cd8fe66ce27f521f70ab
I20260620 04:36:04.982936 7916 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56649
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:36:04.990811 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:04.981615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43035 (local address 127.0.98.62:33601)
0620 04:36:04.981719 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:04.981723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:04.983664 (+ 1941us) server_negotiation.cc:407] Connection header received
0620 04:36:04.983845 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:04.983851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:04.983912 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:04.984003 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:04.984509 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.985159 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.985962 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.986179 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.986278 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:04.987495 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:04.987498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:04.987501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:04.987529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:04.989026 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.989654 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.989659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.989661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.989743 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:04.990020 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:04.990025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:04.990026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:04.990193 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:04.990295 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:04.990554 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 04:36:04.990656 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:36:04.990820 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:04.981806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.1:43035)
0620 04:36:04.983588 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:04.983600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:04.983708 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:04.984032 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:04.984035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:04.984265 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:04.984379 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:04.984386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.985287 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.985290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:04.985847 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:04.985855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:04.986291 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:04.986295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:04.986389 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:04.987028 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:04.987043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:04.987363 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:04.989162 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:04.989165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:04.989167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:04.989381 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:04.989834 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:04.989838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:04.989840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:04.989929 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:04.990296 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:04.990300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:04.990393 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 04:36:04.990502 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 04:36:04.990650 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:36:05.007450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33601
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:05.119706 8129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:05.119966 8129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:05.120009 8129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:05.124243 8129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:05.124302 8129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:05.124341 8129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:36:05.124439 8129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:05.129616 8129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:05.130620 8129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:05.131645 8129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:05.140937 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:05.140937 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:05.140938 8137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:05.140985 8129 server_base.cc:1034] running on GCE node
I20260620 04:36:05.141556 8129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:05.142096 8129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:05.143275 8129 hybrid_clock.cc:648] HybridClock initialized: now 1781930165143243 us; error 44 us; skew 500 ppm
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:05.145963 8129 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:36:05.146857 8129 webserver.cc:460] Webserver started at http://127.0.98.2:34701/ using document root <none> and password file <none>
I20260620 04:36:05.147349 8129 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:05.147404 8129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:05.147620 8129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:05.149564 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dc964df78c54e4d89103eb0f5512fe3"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "52b5c5361987b62527d8d4d175d529ad"
server_key_iv: "52c399a0a8625ba7dac7ffe8e09c52fc"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.150053 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dc964df78c54e4d89103eb0f5512fe3"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "52b5c5361987b62527d8d4d175d529ad"
server_key_iv: "52c399a0a8625ba7dac7ffe8e09c52fc"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.153599 8129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:05.155817 8144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:05.157091 8129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:05.157230 8129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "0dc964df78c54e4d89103eb0f5512fe3"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "52b5c5361987b62527d8d4d175d529ad"
server_key_iv: "52c399a0a8625ba7dac7ffe8e09c52fc"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.157335 8129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:05.183743 8129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:05.186681 8129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:05.186882 8129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:05.188329 8129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:05.188393 8129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:05.188457 8129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:05.188500 8129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:05.199414 8129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39717
I20260620 04:36:05.199481 8259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39717 every 8 connection(s)
I20260620 04:36:05.200413 8129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:05.202397 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.188990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34137 (local address 127.0.98.62:33601)
0620 04:36:05.189139 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.189146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.190119 (+ 973us) server_negotiation.cc:407] Connection header received
0620 04:36:05.191453 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.191457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.191543 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.191641 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.193236 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.193859 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.195642 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.195981 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.196109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.198761 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.198764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.198767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.198791 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.200362 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.200857 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.200863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.200869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.200961 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.201319 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.201323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.201329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.201506 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.201594 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.202113 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.202233 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:36:05.203119 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.189356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.2:34137)
0620 04:36:05.189929 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.189983 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.191215 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.191835 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.191847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.192393 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:05.193056 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.193077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.194009 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.194016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.195508 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.195519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.196150 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.196154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.196350 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.197050 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:05.197066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:05.198533 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:05.200471 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.200476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.200490 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.200756 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.201063 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.201066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.201067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.201225 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.201642 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:05.201647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:05.201850 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.202049 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.202341 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":135,"threads_started":1}
I20260620 04:36:05.203598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8129
I20260620 04:36:05.203724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:36:05.204064 392 external_mini_cluster.cc:1428] Setting key 789fef1c33ad9c0f0df2fefb5fff0387
I20260620 04:36:05.204424 8260 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33601
I20260620 04:36:05.204578 8260 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:05.205099 8260 heartbeater.cc:511] Master 127.0.98.62:33601 requested a full tablet report, sending...
I20260620 04:36:05.206166 7917 ts_manager.cc:194] Registered new tserver with Master: 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717)
I20260620 04:36:05.206856 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34137
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:36:05.214886 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.206477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54535 (local address 127.0.98.62:33601)
0620 04:36:05.206572 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.206576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.207334 (+ 758us) server_negotiation.cc:407] Connection header received
0620 04:36:05.207520 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.207523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.207570 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.207653 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.208216 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.208924 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.210214 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.210429 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.210511 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.211779 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.211781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.211783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.211803 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.213211 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.213724 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.213726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.213727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.213812 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.214093 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.214096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.214097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.214301 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.214382 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.214655 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.214757 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:36:05.214898 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.206426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.2:54535)
0620 04:36:05.207248 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.207266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.207399 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.207683 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.207687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.207910 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:05.208068 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.208080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.209058 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.209064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.210078 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.210091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.210546 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.210561 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.210693 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.211180 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:05.211200 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:05.211639 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:05.213314 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.213317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.213319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.213577 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.213906 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.213908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.213910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.213983 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.214384 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:05.214386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:05.214451 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.214590 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.214720 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:36:05.229350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33601
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:05.335253 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:05.335500 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:05.335541 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:05.339514 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:05.339566 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:05.339604 8269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:36:05.339635 8269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:05.344641 8269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:05.345533 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:05.346410 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:36:05.355574 8269 server_base.cc:1034] running on GCE node
W20260620 04:36:05.355526 8274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:05.355515 8275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:05.355527 8277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:05.356114 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:05.356729 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:05.357937 8269 hybrid_clock.cc:648] HybridClock initialized: now 1781930165357912 us; error 42 us; skew 500 ppm
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:05.360651 8269 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:36:05.361490 8269 webserver.cc:460] Webserver started at http://127.0.98.3:41397/ using document root <none> and password file <none>
I20260620 04:36:05.361981 8269 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:05.362036 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:05.362191 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:05.364091 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "0142800ec9a84ef99e6687223beb13d4"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "91f0ceceeccadffe4cfaa39bbaa0abb0"
server_key_iv: "1aaadb9b0b846ba3937d57438419fbc6"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.364641 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0142800ec9a84ef99e6687223beb13d4"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "91f0ceceeccadffe4cfaa39bbaa0abb0"
server_key_iv: "1aaadb9b0b846ba3937d57438419fbc6"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.368006 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:05.370590 8284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:05.371829 8269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:05.371963 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "0142800ec9a84ef99e6687223beb13d4"
format_stamp: "Formatted at 2026-06-20 04:36:05 on dist-test-slave-zmf4"
server_key: "91f0ceceeccadffe4cfaa39bbaa0abb0"
server_key_iv: "1aaadb9b0b846ba3937d57438419fbc6"
server_key_version: "encryptionkey@0"
I20260620 04:36:05.372082 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:05.386257 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:05.389405 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:05.389675 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:05.391127 8269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:05.391177 8269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:05.391218 8269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:05.391268 8269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:05.402308 8269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33119
I20260620 04:36:05.402408 8399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33119 every 8 connection(s)
I20260620 04:36:05.403304 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:05.404968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8269
I20260620 04:36:05.404995 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.391831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44893 (local address 127.0.98.62:33601)
0620 04:36:05.391955 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.391959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.392993 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 04:36:05.393998 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.394001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.394076 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.394138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.395527 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.396168 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.397351 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.397583 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.397690 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.401201 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.401204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.401207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.401238 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.402809 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.403369 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.403374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.403375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.403461 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.403855 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.403858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.403860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.404097 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.404233 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.404705 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.404825 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:36:05.405123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:05.405488 392 external_mini_cluster.cc:1428] Setting key bbdae4e4c6e0f5d466d089b1908a819a
I20260620 04:36:05.405793 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.392357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.3:44893)
0620 04:36:05.392840 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.392878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.393761 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.394437 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.394443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.394845 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:05.395384 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.395398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.396346 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.396350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.397227 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.397236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.397727 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.397731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.397946 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.398594 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:05.398614 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:05.401007 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:05.402915 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.402924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.402940 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.403255 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.403546 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.403549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.403550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.403734 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.404249 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:05.404256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:05.404475 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.404663 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.404957 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260620 04:36:05.407084 8400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33601
I20260620 04:36:05.407313 8400 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:05.407805 8400 heartbeater.cc:511] Master 127.0.98.62:33601 requested a full tablet report, sending...
I20260620 04:36:05.408907 7917 ts_manager.cc:194] Registered new tserver with Master: 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:05.409611 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44893
I20260620 04:36:05.415910 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.409385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53923 (local address 127.0.98.62:33601)
0620 04:36:05.409508 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.409512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.409739 (+ 227us) server_negotiation.cc:407] Connection header received
0620 04:36:05.409796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.409799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.409843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.409938 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.410463 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.410958 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.411731 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.411955 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.412057 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.413159 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.413162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.413165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.413187 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.414383 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.414804 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.414808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.414810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.414886 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.415152 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.415155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.415157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.415318 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.415415 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.415656 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.415756 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:36:05.415916 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.409327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.98.3:53923)
0620 04:36:05.409609 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.409625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.409727 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.410061 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.410063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.410257 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:05.410352 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.410357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.411074 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.411078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.411622 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.411629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.412059 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.412063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.412155 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.412690 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:05.412708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:05.413023 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:05.414511 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.414516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.414521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.414710 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.414974 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.414977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.414979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.415064 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.415405 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:05.415408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:05.415494 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.415597 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.415730 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":114,"threads_started":1}
I20260620 04:36:05.419305 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:36:05.442348 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:05.436054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.0.98.62:33601)
0620 04:36:05.436299 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.436302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.436314 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:36:05.436351 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.436354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.436390 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.436456 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.437208 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.437721 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.438579 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.438812 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.438902 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.442218 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55298: BlockingRecv error: recv got EOF from 127.0.0.1:55298 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:05.451920 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.443688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.98.62:33601)
0620 04:36:05.443794 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.443797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.443812 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:05.443916 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.443919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.443953 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.444022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.444705 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.445212 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.445996 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.446200 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.446282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.448738 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.448741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.448743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.448764 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.450339 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.450764 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.450767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.450768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.450844 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.451073 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.451080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.451081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.451242 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.451341 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.451699 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.451803 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:36:05.454875 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:36:05.458228 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:05.472041 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.466964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.0.98.3:33119)
0620 04:36:05.467756 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.467766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.467787 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:36:05.467851 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.467854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.468005 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.468118 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:05.468472 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.469331 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.470802 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.471404 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.471521 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.471681 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.471806 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.471894 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":111,"threads_started":1}
I20260620 04:36:05.472030 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.466596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33119 (local address 127.0.0.1:44868)
0620 04:36:05.467447 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.467463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.467529 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.468136 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.468143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.468171 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:05.468340 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.468350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.469425 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.469431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.470681 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.470691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.471527 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.471530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.471625 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.471661 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.471768 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.471881 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":62,"thread_start_us":110,"threads_started":1}
I20260620 04:36:05.472355 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.466534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39717 (local address 127.0.0.1:48234)
0620 04:36:05.467224 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.467275 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.467394 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.468234 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.468237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.468256 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:05.468355 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.468361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.469792 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.469795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.470914 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.470923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.471973 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.471976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.472064 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.472087 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.472133 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.472226 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260620 04:36:05.473106 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.466650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.2:39717)
0620 04:36:05.467806 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.467814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.467832 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:36:05.467891 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.467894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.468068 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.468243 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:05.468512 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.469660 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.471038 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.471848 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.472674 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.472809 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.472856 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.472942 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":95,"threads_started":1}
I20260620 04:36:05.473379 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.467187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.0.1:51856)
0620 04:36:05.468369 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.468381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.468460 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.468925 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.468928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.468945 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:05.469026 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.469031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.470327 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.470330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.471527 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.471537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.472851 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.472854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.472946 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.472966 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.473046 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.473109 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"thread_start_us":84,"threads_started":1}
I20260620 04:36:05.473749 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.467433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.0.98.1:43585)
0620 04:36:05.467707 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.467721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.468545 (+ 824us) server_negotiation.cc:407] Connection header received
0620 04:36:05.468622 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.468627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.468787 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.468926 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:05.469147 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.470219 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.471662 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.472534 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.472643 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.472791 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.473523 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.473610 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260620 04:36:05.475313 8334 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a (DEFAULT_TABLE table=test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.476441 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0bc18f27a4d18963e5bf69264bc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.476219 8054 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a (DEFAULT_TABLE table=test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.477186 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0bc18f27a4d18963e5bf69264bc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.476984 8194 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a (DEFAULT_TABLE table=test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.477981 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0bc18f27a4d18963e5bf69264bc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.483990 8428 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Bootstrap starting.
I20260620 04:36:05.484764 8430 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Bootstrap starting.
I20260620 04:36:05.485394 8429 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Bootstrap starting.
I20260620 04:36:05.486454 8430 tablet_bootstrap.cc:654] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.486663 8428 tablet_bootstrap.cc:654] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.487190 8430 log.cc:824] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:05.487231 8429 tablet_bootstrap.cc:654] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.487421 8428 log.cc:824] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:05.487962 8429 log.cc:824] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:05.489229 8430 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: No bootstrap required, opened a new log
I20260620 04:36:05.489482 8430 ts_tablet_manager.cc:1397] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:36:05.489836 8429 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: No bootstrap required, opened a new log
I20260620 04:36:05.490025 8429 ts_tablet_manager.cc:1397] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:36:05.493291 8428 tablet_bootstrap.cc:492] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: No bootstrap required, opened a new log
I20260620 04:36:05.493553 8428 ts_tablet_manager.cc:1397] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 04:36:05.493574 8430 raft_consensus.cc:348] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.493885 8430 raft_consensus.cc:374] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.493942 8430 raft_consensus.cc:729] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142800ec9a84ef99e6687223beb13d4, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.493803 8429 raft_consensus.cc:348] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.494020 8429 raft_consensus.cc:374] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.494076 8429 raft_consensus.cc:729] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc964df78c54e4d89103eb0f5512fe3, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.494388 8430 consensus_queue.cc:260] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.494508 8429 consensus_queue.cc:260] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.495390 8429 ts_tablet_manager.cc:1428] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:36:05.495390 8430 ts_tablet_manager.cc:1428] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:36:05.495741 8260 heartbeater.cc:503] Master 127.0.98.62:33601 was elected leader, sending a full tablet report...
I20260620 04:36:05.495743 8400 heartbeater.cc:503] Master 127.0.98.62:33601 was elected leader, sending a full tablet report...
I20260620 04:36:05.497139 8428 raft_consensus.cc:348] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.497311 8428 raft_consensus.cc:374] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.497341 8428 raft_consensus.cc:729] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.497699 8428 consensus_queue.cc:260] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.498198 8120 heartbeater.cc:503] Master 127.0.98.62:33601 was elected leader, sending a full tablet report...
I20260620 04:36:05.498636 8428 ts_tablet_manager.cc:1428] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 04:36:05.654281 8401 tablet.cc:2369] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:36:05.701692 8261 tablet.cc:2369] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:05.720880 8436 raft_consensus.cc:482] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:05.721136 8436 raft_consensus.cc:504] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.722522 8436 leader_election.cc:288] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:05.726687 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.722798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33119 (local address 127.0.98.1:53433)
0620 04:36:05.723123 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.723150 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.723257 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.723508 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.723511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.723527 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:05.723662 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.723669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.724560 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.724563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.725467 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.725477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.726230 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.726234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.726346 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.726383 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.726453 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.726524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260620 04:36:05.726714 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.722924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53433 (local address 127.0.98.3:33119)
0620 04:36:05.723068 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.723075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.723233 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:36:05.723352 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.723356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.723412 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.723500 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:05.723778 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.724458 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.725595 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.726102 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.726199 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.726251 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.726486 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.726561 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:36:05.726887 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.723006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33337 (local address 127.0.98.2:39717)
0620 04:36:05.723125 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.723130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.723368 (+ 238us) server_negotiation.cc:407] Connection header received
0620 04:36:05.723505 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.723508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.723554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.723639 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:05.723959 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.724655 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.725837 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.726284 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.726387 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.726461 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.726675 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.726761 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:36:05.726903 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.722798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39717 (local address 127.0.98.1:33337)
0620 04:36:05.723259 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.723276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.723395 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.723669 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.723672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.723686 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:05.723822 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.723830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.724769 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.724771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.725691 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.725701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.726406 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.726409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.726543 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.726567 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.726660 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.726730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":50,"thread_start_us":100,"threads_started":1}
I20260620 04:36:05.727306 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0bc18f27a4d18963e5bf69264bc4a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4" is_pre_election: true
I20260620 04:36:05.727511 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0bc18f27a4d18963e5bf69264bc4a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3" is_pre_election: true
I20260620 04:36:05.727610 8354 raft_consensus.cc:2454] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:05.727777 8214 raft_consensus.cc:2454] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:05.728121 8005 leader_election.cc:302] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:05.728416 8436 raft_consensus.cc:2781] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:05.728500 8436 raft_consensus.cc:482] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:05.728554 8436 raft_consensus.cc:3037] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.729601 8436 raft_consensus.cc:504] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260620 04:36:05.729969 8436 leader_election.cc:288] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Requested vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:05.730244 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0bc18f27a4d18963e5bf69264bc4a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3"
I20260620 04:36:05.730373 8214 raft_consensus.cc:3037] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.730356 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0bc18f27a4d18963e5bf69264bc4a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4"
I20260620 04:36:05.730515 8354 raft_consensus.cc:3037] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.731640 8214 raft_consensus.cc:2454] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:05.731933 8006 leader_election.cc:302] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc964df78c54e4d89103eb0f5512fe3, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:05.732112 8436 raft_consensus.cc:2781] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:05.732157 8354 raft_consensus.cc:2454] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:05.732340 8436 raft_consensus.cc:686] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Becoming Leader. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Running, Role: LEADER
I20260620 04:36:05.732653 8436 consensus_queue.cc:237] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
W20260620 04:36:05.734484 8121 tablet.cc:2369] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:05.735733 7917 catalog_manager.cc:5495] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16704c72df034068816e50e2ea2bf41f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: HEALTHY } } }
I20260620 04:36:05.762802 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.759356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.98.1:43585)
0620 04:36:05.759483 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.759488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.759503 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:05.759609 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.759611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.759671 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.759773 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:05.760014 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.760553 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.761450 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.761667 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.761765 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.762013 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:05.762427 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:05.762545 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.762606 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.762666 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:36:05.768255 8354 raft_consensus.cc:1261] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:05.768265 8214 raft_consensus.cc:1261] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:05.768904 8439 consensus_queue.cc:1035] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:05.769089 8436 consensus_queue.cc:1035] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:05.777864 8443 mvcc.cc:205] Tried to move back new op lower bound from 7298785958979977216 to 7298785958843629568. Current Snapshot: MvccSnapshot[applied={T|T < 7298785958979977216}]
I20260620 04:36:05.784178 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:05.784484 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:05.786582 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Sending DeleteTablet for 3 replicas of tablet e0e0bc18f27a4d18963e5bf69264bc4a
I20260620 04:36:05.787353 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 04:36:05.787360 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260620 04:36:05.787809 8453 tablet_replica.cc:330] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: stopping tablet replica
I20260620 04:36:05.787847 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:36:05.787922 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e0bc18f27a4d18963e5bf69264bc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48234
I20260620 04:36:05.788100 8453 raft_consensus.cc:2229] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 04:36:05.788257 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:05.788407 8453 raft_consensus.cc:2258] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:05.788638 8454 tablet_replica.cc:330] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: stopping tablet replica
I20260620 04:36:05.788861 8452 tablet_replica.cc:330] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: stopping tablet replica
I20260620 04:36:05.789170 8452 raft_consensus.cc:2229] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:05.789618 8454 raft_consensus.cc:2229] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:05.789916 8453 ts_tablet_manager.cc:1905] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:05.790810 8452 raft_consensus.cc:2258] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:05.791253 8454 raft_consensus.cc:2258] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:05.792670 8454 ts_tablet_manager.cc:1905] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:05.792846 8453 ts_tablet_manager.cc:1918] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:05.792938 8453 log.cc:1196] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/e0e0bc18f27a4d18963e5bf69264bc4a
I20260620 04:36:05.793232 8453 ts_tablet_manager.cc:1939] T e0e0bc18f27a4d18963e5bf69264bc4a P 0142800ec9a84ef99e6687223beb13d4: Deleting consensus metadata
I20260620 04:36:05.794217 8334 tablet_service.cc:1467] Processing CreateTablet for tablet fa0fe79229574855b65559dbd156438a (DEFAULT_TABLE table=test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.794474 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0fe79229574855b65559dbd156438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.794493 8194 tablet_service.cc:1467] Processing CreateTablet for tablet fa0fe79229574855b65559dbd156438a (DEFAULT_TABLE table=test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.795040 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0fe79229574855b65559dbd156438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.795553 8452 ts_tablet_manager.cc:1905] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:05.795622 8054 tablet_service.cc:1467] Processing CreateTablet for tablet fa0fe79229574855b65559dbd156438a (DEFAULT_TABLE table=test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:05.795882 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0fe79229574855b65559dbd156438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.797385 7901 catalog_manager.cc:4841] TS 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119): tablet e0e0bc18f27a4d18963e5bf69264bc4a (table test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]) successfully deleted
I20260620 04:36:05.798038 8452 ts_tablet_manager.cc:1918] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:05.798126 8452 log.cc:1196] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/e0e0bc18f27a4d18963e5bf69264bc4a
I20260620 04:36:05.798169 8428 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Bootstrap starting.
I20260620 04:36:05.798259 8430 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Bootstrap starting.
I20260620 04:36:05.798410 8452 ts_tablet_manager.cc:1939] T e0e0bc18f27a4d18963e5bf69264bc4a P 16704c72df034068816e50e2ea2bf41f: Deleting consensus metadata
I20260620 04:36:05.799140 8428 tablet_bootstrap.cc:654] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.799140 8430 tablet_bootstrap.cc:654] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.799892 7902 catalog_manager.cc:4841] TS 16704c72df034068816e50e2ea2bf41f (127.0.98.1:43585): tablet e0e0bc18f27a4d18963e5bf69264bc4a (table test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]) successfully deleted
I20260620 04:36:05.800369 8430 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: No bootstrap required, opened a new log
I20260620 04:36:05.800444 8430 ts_tablet_manager.cc:1397] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:05.800853 8430 raft_consensus.cc:348] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.800982 8430 raft_consensus.cc:374] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.801014 8430 raft_consensus.cc:729] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142800ec9a84ef99e6687223beb13d4, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.801143 8430 consensus_queue.cc:260] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.801434 8430 ts_tablet_manager.cc:1428] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:05.802204 8429 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Bootstrap starting.
I20260620 04:36:05.803203 8429 tablet_bootstrap.cc:654] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.804968 8429 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: No bootstrap required, opened a new log
I20260620 04:36:05.805063 8429 ts_tablet_manager.cc:1397] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:05.805099 8454 ts_tablet_manager.cc:1918] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:05.805193 8454 log.cc:1196] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/e0e0bc18f27a4d18963e5bf69264bc4a
I20260620 04:36:05.805516 8454 ts_tablet_manager.cc:1939] T e0e0bc18f27a4d18963e5bf69264bc4a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting consensus metadata
I20260620 04:36:05.805585 8429 raft_consensus.cc:348] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.805720 8429 raft_consensus.cc:374] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.805759 8429 raft_consensus.cc:729] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc964df78c54e4d89103eb0f5512fe3, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.806175 8429 consensus_queue.cc:260] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.806509 8429 ts_tablet_manager.cc:1428] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:05.806701 7902 catalog_manager.cc:4841] TS 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717): tablet e0e0bc18f27a4d18963e5bf69264bc4a (table test-table [id=0ef129fbfe0d47e5b5c5737e4c786b98]) successfully deleted
I20260620 04:36:05.807688 8428 tablet_bootstrap.cc:492] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: No bootstrap required, opened a new log
I20260620 04:36:05.807782 8428 ts_tablet_manager.cc:1397] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 04:36:05.808326 8428 raft_consensus.cc:348] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.808454 8428 raft_consensus.cc:374] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.808498 8428 raft_consensus.cc:729] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.808624 8428 consensus_queue.cc:260] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.808898 8428 ts_tablet_manager.cc:1428] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:05.909232 8448 raft_consensus.cc:482] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:05.909390 8448 raft_consensus.cc:504] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.909883 8448 leader_election.cc:288] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:05.910321 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0fe79229574855b65559dbd156438a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3" is_pre_election: true
I20260620 04:36:05.910322 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0fe79229574855b65559dbd156438a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4" is_pre_election: true
I20260620 04:36:05.910477 8354 raft_consensus.cc:2454] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:05.910481 8214 raft_consensus.cc:2454] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:05.910790 8005 leader_election.cc:302] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:05.911006 8448 raft_consensus.cc:2781] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:05.911065 8448 raft_consensus.cc:482] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:05.911096 8448 raft_consensus.cc:3037] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.912132 8448 raft_consensus.cc:504] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.912549 8448 leader_election.cc:288] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Requested vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:05.912856 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0fe79229574855b65559dbd156438a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3"
I20260620 04:36:05.912958 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0fe79229574855b65559dbd156438a" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4"
I20260620 04:36:05.913072 8354 raft_consensus.cc:3037] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.913075 8214 raft_consensus.cc:3037] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:05.914072 8214 raft_consensus.cc:2454] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:05.914248 8354 raft_consensus.cc:2454] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:05.914368 8006 leader_election.cc:302] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc964df78c54e4d89103eb0f5512fe3, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:05.914660 8448 raft_consensus.cc:2781] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:05.914840 8448 raft_consensus.cc:686] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Becoming Leader. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Running, Role: LEADER
I20260620 04:36:05.914997 8448 consensus_queue.cc:237] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.917232 7916 catalog_manager.cc:5495] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16704c72df034068816e50e2ea2bf41f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:36:05 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:05.959187 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.949809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33601 (local address 127.0.0.1:55322)
0620 04:36:05.950136 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:05.950156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:05.950261 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:05.950599 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:05.950602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:05.951001 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:05.951097 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.951103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.951899 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.951905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.952709 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:05.952718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.953123 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.953126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:05.953207 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.953750 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:05.953762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:05.955131 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:05.957269 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.957297 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.957305 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.957740 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.958060 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:05.958063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:05.958065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:05.958188 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:05.958643 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:05.958652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:05.958787 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 04:36:05.958909 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 04:36:05.959026 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":131,"threads_started":1}
I20260620 04:36:05.959205 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:05.949894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.98.62:33601)
0620 04:36:05.950035 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:05.950039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:05.950244 (+ 205us) server_negotiation.cc:407] Connection header received
0620 04:36:05.950385 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:05.950388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:05.950429 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:05.950518 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:05.951271 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.951777 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.952819 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:05.953023 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:05.953123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:05.955276 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:05.955279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:05.955281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:05.955303 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:05.957176 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.957849 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.957852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.957855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.957941 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:05.958300 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:05.958303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:05.958305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:05.958524 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:05.958621 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:05.958992 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 04:36:05.959088 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:36:05.963287 7916 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:36:05.964056 7916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:05.969200 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 20ddfffa97bb4435839b476ef266a03b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ed3fecc2d7a4090aac4292a59de1570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:36:05.969177 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 20ddfffa97bb4435839b476ef266a03b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ed3fecc2d7a4090aac4292a59de1570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:36:05.969177 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 20ddfffa97bb4435839b476ef266a03b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ed3fecc2d7a4090aac4292a59de1570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:36:05.969575 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ddfffa97bb4435839b476ef266a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.969581 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ddfffa97bb4435839b476ef266a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.969595 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ddfffa97bb4435839b476ef266a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:05.972007 8430 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: Bootstrap starting.
I20260620 04:36:05.972514 8428 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: Bootstrap starting.
I20260620 04:36:05.972534 8429 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: Bootstrap starting.
I20260620 04:36:05.973071 8430 tablet_bootstrap.cc:654] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.973510 8428 tablet_bootstrap.cc:654] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.973690 8429 tablet_bootstrap.cc:654] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:05.974503 8430 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: No bootstrap required, opened a new log
I20260620 04:36:05.974586 8430 ts_tablet_manager.cc:1397] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:05.974730 8428 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: No bootstrap required, opened a new log
I20260620 04:36:05.974807 8428 ts_tablet_manager.cc:1397] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:05.975117 8430 raft_consensus.cc:348] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.975224 8430 raft_consensus.cc:374] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.975258 8430 raft_consensus.cc:729] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142800ec9a84ef99e6687223beb13d4, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.975229 8428 raft_consensus.cc:348] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.975338 8428 raft_consensus.cc:374] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.975378 8428 raft_consensus.cc:729] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.975385 8430 consensus_queue.cc:260] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.975494 8428 consensus_queue.cc:260] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.975610 8429 tablet_bootstrap.cc:492] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: No bootstrap required, opened a new log
I20260620 04:36:05.975632 8430 ts_tablet_manager.cc:1428] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:05.975762 8429 ts_tablet_manager.cc:1397] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:05.975864 8428 ts_tablet_manager.cc:1428] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:05.976078 8448 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:05.976197 8448 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:05.976263 8429 raft_consensus.cc:348] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.976358 8429 raft_consensus.cc:374] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:05.976389 8429 raft_consensus.cc:729] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc964df78c54e4d89103eb0f5512fe3, State: Initialized, Role: FOLLOWER
I20260620 04:36:05.976498 8429 consensus_queue.cc:260] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:05.976768 8429 ts_tablet_manager.cc:1428] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:05.976935 8435 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:05.977087 8435 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:05.977458 8434 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:05.977615 8434 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:06.140342 8448 raft_consensus.cc:482] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:06.140561 8448 raft_consensus.cc:504] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.141137 8448 leader_election.cc:288] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.141562 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ddfffa97bb4435839b476ef266a03b" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3" is_pre_election: true
I20260620 04:36:06.141726 8214 raft_consensus.cc:2454] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.141649 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ddfffa97bb4435839b476ef266a03b" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4" is_pre_election: true
I20260620 04:36:06.141847 8354 raft_consensus.cc:2454] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.142025 8006 leader_election.cc:302] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc964df78c54e4d89103eb0f5512fe3, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.142206 8448 raft_consensus.cc:2781] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:06.142279 8448 raft_consensus.cc:482] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:06.142309 8448 raft_consensus.cc:3037] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.143332 8448 raft_consensus.cc:504] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.143654 8448 leader_election.cc:288] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Requested vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.143982 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ddfffa97bb4435839b476ef266a03b" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3"
I20260620 04:36:06.143996 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ddfffa97bb4435839b476ef266a03b" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4"
I20260620 04:36:06.144116 8214 raft_consensus.cc:3037] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.144133 8354 raft_consensus.cc:3037] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.145210 8354 raft_consensus.cc:2454] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.145210 8214 raft_consensus.cc:2454] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.145498 8005 leader_election.cc:302] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.145687 8448 raft_consensus.cc:2781] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:06.145809 8448 raft_consensus.cc:686] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Becoming Leader. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Running, Role: LEADER
I20260620 04:36:06.145942 8448 consensus_queue.cc:237] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.146752 8439 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: TxnStatusTablet state changed. Reason: New leader 16704c72df034068816e50e2ea2bf41f. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.146862 8439 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:36:06.147171 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:36:06.147348 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:36:06.147922 7914 catalog_manager.cc:5495] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16704c72df034068816e50e2ea2bf41f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } health_report { overall_health: UNKNOWN } } }
I20260620 04:36:06.171820 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.168870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.98.1:43585)
0620 04:36:06.169016 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:06.169020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:06.169037 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:36:06.169095 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:06.169098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:06.169147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:06.169222 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:06.169465 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.169957 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.170785 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.171007 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.171112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.171374 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:06.171441 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:06.171517 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:06.171634 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 04:36:06.171683 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:36:06.171820 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.168719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.0.1:51880)
0620 04:36:06.168863 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:06.168881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:06.168987 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:06.169252 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:06.169255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:06.169265 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:36:06.169366 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:06.169371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.170066 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.170071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:06.170671 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:06.170678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.171111 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.171115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:06.171201 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.171241 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:36:06.171520 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:36:06.171536 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:36:06.171613 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:36:06.171701 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 04:36:06.174367 8214 raft_consensus.cc:1261] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.174429 8354 raft_consensus.cc:1261] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.174880 8439 consensus_queue.cc:1035] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.175231 8439 consensus_queue.cc:1035] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.176661 8434 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: TxnStatusTablet state changed. Reason: New leader 16704c72df034068816e50e2ea2bf41f. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.176774 8434 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:06.177209 8435 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: TxnStatusTablet state changed. Reason: New leader 16704c72df034068816e50e2ea2bf41f. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.177306 8435 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:06.178923 8448 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.179019 8448 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:36:06.179121 8448 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.179191 8448 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 16704c72df034068816e50e2ea2bf41f: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:36:06.179270 8442 mvcc.cc:205] Tried to move back new op lower bound from 7298785960646111232 to 7298785960535777280. Current Snapshot: MvccSnapshot[applied={T|T < 7298785960646111232}]
I20260620 04:36:06.179338 8444 mvcc.cc:205] Tried to move back new op lower bound from 7298785960646111232 to 7298785960535777280. Current Snapshot: MvccSnapshot[applied={T|T < 7298785960646111232}]
I20260620 04:36:06.179454 8434 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.179543 8434 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0142800ec9a84ef99e6687223beb13d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:06.179911 8435 tablet_replica.cc:439] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } } }
I20260620 04:36:06.180009 8435 tablet_replica.cc:442] T 20ddfffa97bb4435839b476ef266a03b P 0dc964df78c54e4d89103eb0f5512fe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:36:06.194077 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.190239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.98.1:47561)
0620 04:36:06.190496 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:06.190509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:06.190628 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:06.190833 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:06.190836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:06.190843 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:36:06.190934 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:06.190940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.191622 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.191628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:06.192274 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:06.192283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.193068 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.193071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:06.193164 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.193309 (+ 145us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:36:06.193792 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:36:06.193799 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:36:06.193863 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:36:06.193978 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260620 04:36:06.194077 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.190304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47561 (local address 127.0.98.1:43585)
0620 04:36:06.190425 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:06.190430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:06.190588 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:36:06.190708 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:06.190712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:06.190753 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:06.190828 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:06.191038 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.191513 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.192465 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.192920 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.193070 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.193637 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:06.193711 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:06.193790 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:06.193871 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 04:36:06.193938 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":46}
I20260620 04:36:06.199985 8354 raft_consensus.cc:1261] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.200031 8214 raft_consensus.cc:1261] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.200428 8439 consensus_queue.cc:1035] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.200606 8448 consensus_queue.cc:1035] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.241631 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.241794 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.243968 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Sending DeleteTablet for 3 replicas of tablet fa0fe79229574855b65559dbd156438a
I20260620 04:36:06.244563 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0fe79229574855b65559dbd156438a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 04:36:06.244616 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0fe79229574855b65559dbd156438a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48234
I20260620 04:36:06.244733 8452 tablet_replica.cc:330] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: stopping tablet replica
I20260620 04:36:06.244817 8454 tablet_replica.cc:330] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: stopping tablet replica
I20260620 04:36:06.244853 8452 raft_consensus.cc:2229] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:06.244942 8454 raft_consensus.cc:2229] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.245054 8452 raft_consensus.cc:2258] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.245064 8454 raft_consensus.cc:2258] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.245124 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0fe79229574855b65559dbd156438a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260620 04:36:06.245301 8453 tablet_replica.cc:330] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: stopping tablet replica
I20260620 04:36:06.245431 8453 raft_consensus.cc:2229] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.245545 8453 raft_consensus.cc:2258] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.246263 8452 ts_tablet_manager.cc:1905] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.246263 8454 ts_tablet_manager.cc:1905] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.249101 8453 ts_tablet_manager.cc:1905] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.249259 8452 ts_tablet_manager.cc:1918] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:36:06.249259 8454 ts_tablet_manager.cc:1918] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:36:06.249333 8452 log.cc:1196] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/fa0fe79229574855b65559dbd156438a
I20260620 04:36:06.249332 8454 log.cc:1196] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/fa0fe79229574855b65559dbd156438a
I20260620 04:36:06.249624 8452 ts_tablet_manager.cc:1939] T fa0fe79229574855b65559dbd156438a P 16704c72df034068816e50e2ea2bf41f: Deleting consensus metadata
I20260620 04:36:06.249616 8454 ts_tablet_manager.cc:1939] T fa0fe79229574855b65559dbd156438a P 0dc964df78c54e4d89103eb0f5512fe3: Deleting consensus metadata
I20260620 04:36:06.250313 7902 catalog_manager.cc:4841] TS 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717): tablet fa0fe79229574855b65559dbd156438a (table test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]) successfully deleted
I20260620 04:36:06.250468 7902 catalog_manager.cc:4841] TS 16704c72df034068816e50e2ea2bf41f (127.0.98.1:43585): tablet fa0fe79229574855b65559dbd156438a (table test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]) successfully deleted
I20260620 04:36:06.251775 8453 ts_tablet_manager.cc:1918] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:36:06.251853 8453 log.cc:1196] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/fa0fe79229574855b65559dbd156438a
I20260620 04:36:06.252091 8453 ts_tablet_manager.cc:1939] T fa0fe79229574855b65559dbd156438a P 0142800ec9a84ef99e6687223beb13d4: Deleting consensus metadata
I20260620 04:36:06.252782 7901 catalog_manager.cc:4841] TS 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119): tablet fa0fe79229574855b65559dbd156438a (table test-table [id=687e5024bd9d4d78a72d4fe1b068f5cb]) successfully deleted
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:36:06.278090 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:06.270781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.98.62:33601)
0620 04:36:06.270929 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:06.270935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:06.271111 (+ 176us) server_negotiation.cc:407] Connection header received
0620 04:36:06.271240 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:06.271243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:06.271286 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:06.271356 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:06.272274 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.273232 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.274177 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.274438 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.274559 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.277925 (+ 3366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55332: BlockingRecv error: recv got EOF from 127.0.0.1:55332 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:36:06.289528 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.279472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.0.98.62:33601)
0620 04:36:06.279595 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:06.279600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:06.279997 (+ 397us) server_negotiation.cc:407] Connection header received
0620 04:36:06.280115 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:06.280118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:06.280163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:06.280286 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:06.280884 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.281659 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.282673 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.282997 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.283132 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.286037 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:06.286041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:06.286044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:06.286082 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:06.287784 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:06.288236 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:06.288239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:06.288241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:06.288331 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:06.288624 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:06.288628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:06.288629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:06.288858 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:06.288997 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:06.289253 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 04:36:06.289367 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:36:06.292510 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:36:06.292973 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:06.298501 8053 tablet_service.cc:1467] Processing CreateTablet for tablet 6b12029410f14e538bc782e9a628e098 (DEFAULT_TABLE table=test-table [id=24a6e4392ecd427db647ed700b0a23b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.298609 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 6b12029410f14e538bc782e9a628e098 (DEFAULT_TABLE table=test-table [id=24a6e4392ecd427db647ed700b0a23b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.298830 8053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b12029410f14e538bc782e9a628e098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.298805 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 6b12029410f14e538bc782e9a628e098 (DEFAULT_TABLE table=test-table [id=24a6e4392ecd427db647ed700b0a23b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.298885 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b12029410f14e538bc782e9a628e098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.299046 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b12029410f14e538bc782e9a628e098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.301321 8429 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Bootstrap starting.
I20260620 04:36:06.301322 8430 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Bootstrap starting.
I20260620 04:36:06.302181 8430 tablet_bootstrap.cc:654] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.302181 8429 tablet_bootstrap.cc:654] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.302801 8428 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Bootstrap starting.
I20260620 04:36:06.303493 8428 tablet_bootstrap.cc:654] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.304772 8428 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: No bootstrap required, opened a new log
I20260620 04:36:06.304857 8428 ts_tablet_manager.cc:1397] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:06.305336 8429 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: No bootstrap required, opened a new log
I20260620 04:36:06.305316 8428 raft_consensus.cc:348] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.305405 8428 raft_consensus.cc:374] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.305408 8429 ts_tablet_manager.cc:1397] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 04:36:06.305428 8428 raft_consensus.cc:729] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.305557 8428 consensus_queue.cc:260] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.305814 8428 ts_tablet_manager.cc:1428] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:06.305799 8429 raft_consensus.cc:348] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.305895 8429 raft_consensus.cc:374] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.305922 8429 raft_consensus.cc:729] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc964df78c54e4d89103eb0f5512fe3, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.306023 8429 consensus_queue.cc:260] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.306274 8429 ts_tablet_manager.cc:1428] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:06.306568 8430 tablet_bootstrap.cc:492] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: No bootstrap required, opened a new log
I20260620 04:36:06.306651 8430 ts_tablet_manager.cc:1397] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 04:36:06.307119 8430 raft_consensus.cc:348] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.307226 8430 raft_consensus.cc:374] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.307260 8430 raft_consensus.cc:729] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142800ec9a84ef99e6687223beb13d4, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.307377 8430 consensus_queue.cc:260] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.307690 8430 ts_tablet_manager.cc:1428] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:36:06.425284 8439 raft_consensus.cc:482] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:06.425452 8439 raft_consensus.cc:504] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.425956 8439 leader_election.cc:288] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.426522 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b12029410f14e538bc782e9a628e098" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3" is_pre_election: true
I20260620 04:36:06.426539 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b12029410f14e538bc782e9a628e098" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4" is_pre_election: true
I20260620 04:36:06.426687 8354 raft_consensus.cc:2454] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.426683 8214 raft_consensus.cc:2454] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.427024 8005 leader_election.cc:302] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.427222 8439 raft_consensus.cc:2781] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:06.427291 8439 raft_consensus.cc:482] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:06.427320 8439 raft_consensus.cc:3037] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.428367 8439 raft_consensus.cc:504] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.428714 8439 leader_election.cc:288] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Requested vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.428996 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b12029410f14e538bc782e9a628e098" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3"
I20260620 04:36:06.429070 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b12029410f14e538bc782e9a628e098" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4"
I20260620 04:36:06.429169 8214 raft_consensus.cc:3037] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.429172 8354 raft_consensus.cc:3037] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.430286 8354 raft_consensus.cc:2454] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.430286 8214 raft_consensus.cc:2454] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.430583 8005 leader_election.cc:302] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.430776 8439 raft_consensus.cc:2781] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:06.430878 8439 raft_consensus.cc:686] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Becoming Leader. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Running, Role: LEADER
I20260620 04:36:06.431046 8439 consensus_queue.cc:237] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.432847 7914 catalog_manager.cc:5495] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16704c72df034068816e50e2ea2bf41f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } health_report { overall_health: UNKNOWN } } }
I20260620 04:36:06.463119 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.459841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.0.98.1:43585)
0620 04:36:06.459960 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:06.459964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:06.459977 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:06.460039 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:06.460041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:06.460085 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:06.460153 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:06.460405 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.460975 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.462065 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:06.462305 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:06.462399 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:06.462708 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:06.462781 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:06.462859 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:06.462960 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 04:36:06.463009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:36:06.465159 8354 raft_consensus.cc:1261] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.465248 8214 raft_consensus.cc:1261] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.465644 8439 consensus_queue.cc:1035] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.465811 8448 consensus_queue.cc:1035] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.480597 7915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.480997 7915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.483467 7915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Sending DeleteTablet for 3 replicas of tablet 6b12029410f14e538bc782e9a628e098
I20260620 04:36:06.484768 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b12029410f14e538bc782e9a628e098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260620 04:36:06.484766 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b12029410f14e538bc782e9a628e098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48234
I20260620 04:36:06.484946 8453 tablet_replica.cc:330] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: stopping tablet replica
I20260620 04:36:06.484947 8454 tablet_replica.cc:330] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: stopping tablet replica
I20260620 04:36:06.485060 8453 raft_consensus.cc:2229] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.485060 8454 raft_consensus.cc:2229] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.485191 8453 raft_consensus.cc:2258] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.485162 8333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b12029410f14e538bc782e9a628e098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260620 04:36:06.485522 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b12029410f14e538bc782e9a628e098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 04:36:06.485628 8454 raft_consensus.cc:2258] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.485811 8452 tablet_replica.cc:330] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: stopping tablet replica
I20260620 04:36:06.485903 8453 ts_tablet_manager.cc:1905] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.485929 8452 raft_consensus.cc:2229] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:06.486140 8452 raft_consensus.cc:2258] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.486671 7915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:36:06.487072 8452 ts_tablet_manager.cc:1905] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:36:06.487107 7915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:06.487182 8454 ts_tablet_manager.cc:1905] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.488412 8453 ts_tablet_manager.cc:1918] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:06.488492 8453 log.cc:1196] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/6b12029410f14e538bc782e9a628e098
I20260620 04:36:06.488750 8453 ts_tablet_manager.cc:1939] T 6b12029410f14e538bc782e9a628e098 P 0142800ec9a84ef99e6687223beb13d4: Deleting consensus metadata
I20260620 04:36:06.489500 8452 ts_tablet_manager.cc:1918] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:06.489571 8452 log.cc:1196] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/6b12029410f14e538bc782e9a628e098
I20260620 04:36:06.489818 8452 ts_tablet_manager.cc:1939] T 6b12029410f14e538bc782e9a628e098 P 16704c72df034068816e50e2ea2bf41f: Deleting consensus metadata
I20260620 04:36:06.489802 8454 ts_tablet_manager.cc:1918] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:06.489871 8454 log.cc:1196] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/6b12029410f14e538bc782e9a628e098
I20260620 04:36:06.490135 8454 ts_tablet_manager.cc:1939] T 6b12029410f14e538bc782e9a628e098 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting consensus metadata
I20260620 04:36:06.490936 7901 catalog_manager.cc:4841] TS 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119): tablet 6b12029410f14e538bc782e9a628e098 (table test-table [id=24a6e4392ecd427db647ed700b0a23b2]) successfully deleted
W20260620 04:36:06.491276 7901 catalog_manager.cc:4808] TS 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119): delete failed for tablet 6b12029410f14e538bc782e9a628e098 because the tablet was not found. No further retry: Not found: Tablet not found: 6b12029410f14e538bc782e9a628e098
I20260620 04:36:06.492152 7902 catalog_manager.cc:4841] TS 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717): tablet 6b12029410f14e538bc782e9a628e098 (table test-table [id=24a6e4392ecd427db647ed700b0a23b2]) successfully deleted
I20260620 04:36:06.492269 8333 tablet_service.cc:1467] Processing CreateTablet for tablet ac6d6e01a50746dc8037b9394284ea33 (DEFAULT_TABLE table=test-table [id=e1523b5347c4423f85525500c6299e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.492321 7902 catalog_manager.cc:4841] TS 16704c72df034068816e50e2ea2bf41f (127.0.98.1:43585): tablet 6b12029410f14e538bc782e9a628e098 (table test-table [id=24a6e4392ecd427db647ed700b0a23b2]) successfully deleted
I20260620 04:36:06.492527 8333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6d6e01a50746dc8037b9394284ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.492762 8053 tablet_service.cc:1467] Processing CreateTablet for tablet ac6d6e01a50746dc8037b9394284ea33 (DEFAULT_TABLE table=test-table [id=e1523b5347c4423f85525500c6299e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.492990 8053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6d6e01a50746dc8037b9394284ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.495368 8430 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Bootstrap starting.
I20260620 04:36:06.495517 8194 tablet_service.cc:1467] Processing CreateTablet for tablet ac6d6e01a50746dc8037b9394284ea33 (DEFAULT_TABLE table=test-table [id=e1523b5347c4423f85525500c6299e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:06.495587 8428 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: Bootstrap starting.
I20260620 04:36:06.495761 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6d6e01a50746dc8037b9394284ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:06.496429 8428 tablet_bootstrap.cc:654] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.496464 8430 tablet_bootstrap.cc:654] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.497852 8428 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: No bootstrap required, opened a new log
I20260620 04:36:06.497882 8430 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: No bootstrap required, opened a new log
I20260620 04:36:06.497946 8428 ts_tablet_manager.cc:1397] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:06.497974 8430 ts_tablet_manager.cc:1397] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:06.498451 8429 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Bootstrap starting.
I20260620 04:36:06.498420 8430 raft_consensus.cc:348] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.498457 8428 raft_consensus.cc:348] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.498543 8430 raft_consensus.cc:374] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.498579 8430 raft_consensus.cc:729] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142800ec9a84ef99e6687223beb13d4, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.498584 8428 raft_consensus.cc:374] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.498624 8428 raft_consensus.cc:729] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.498703 8430 consensus_queue.cc:260] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.498768 8428 consensus_queue.cc:260] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.499104 8428 ts_tablet_manager.cc:1428] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:06.499455 8429 tablet_bootstrap.cc:654] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:06.500438 8430 ts_tablet_manager.cc:1428] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:36:06.500954 8429 tablet_bootstrap.cc:492] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: No bootstrap required, opened a new log
I20260620 04:36:06.501056 8429 ts_tablet_manager.cc:1397] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:06.501561 8429 raft_consensus.cc:348] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.501708 8429 raft_consensus.cc:374] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:06.501752 8429 raft_consensus.cc:729] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc964df78c54e4d89103eb0f5512fe3, State: Initialized, Role: FOLLOWER
I20260620 04:36:06.501899 8429 consensus_queue.cc:260] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.502142 8429 ts_tablet_manager.cc:1428] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:36:06.635496 8439 raft_consensus.cc:482] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:06.635673 8439 raft_consensus.cc:504] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.636159 8439 leader_election.cc:288] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.636686 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6d6e01a50746dc8037b9394284ea33" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4" is_pre_election: true
I20260620 04:36:06.636857 8354 raft_consensus.cc:2454] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.636814 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6d6e01a50746dc8037b9394284ea33" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3" is_pre_election: true
I20260620 04:36:06.636994 8214 raft_consensus.cc:2454] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 0.
I20260620 04:36:06.637262 8005 leader_election.cc:302] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142800ec9a84ef99e6687223beb13d4, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.637447 8439 raft_consensus.cc:2781] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:06.637524 8439 raft_consensus.cc:482] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:06.637579 8439 raft_consensus.cc:3037] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.638564 8439 raft_consensus.cc:504] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.638931 8439 leader_election.cc:288] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Requested vote from peers 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717), 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119)
I20260620 04:36:06.639250 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6d6e01a50746dc8037b9394284ea33" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc964df78c54e4d89103eb0f5512fe3"
I20260620 04:36:06.639384 8214 raft_consensus.cc:3037] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.639420 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6d6e01a50746dc8037b9394284ea33" candidate_uuid: "16704c72df034068816e50e2ea2bf41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142800ec9a84ef99e6687223beb13d4"
I20260620 04:36:06.639570 8354 raft_consensus.cc:3037] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:06.640367 8214 raft_consensus.cc:2454] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.640543 8354 raft_consensus.cc:2454] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16704c72df034068816e50e2ea2bf41f in term 1.
I20260620 04:36:06.640640 8006 leader_election.cc:302] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc964df78c54e4d89103eb0f5512fe3, 16704c72df034068816e50e2ea2bf41f; no voters:
I20260620 04:36:06.640813 8439 raft_consensus.cc:2781] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:06.640913 8439 raft_consensus.cc:686] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Becoming Leader. State: Replica: 16704c72df034068816e50e2ea2bf41f, State: Running, Role: LEADER
I20260620 04:36:06.641049 8439 consensus_queue.cc:237] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } }
I20260620 04:36:06.642729 7914 catalog_manager.cc:5495] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f reported cstate change: term changed from 0 to 1, leader changed from <none> to 16704c72df034068816e50e2ea2bf41f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16704c72df034068816e50e2ea2bf41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16704c72df034068816e50e2ea2bf41f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 } health_report { overall_health: UNKNOWN } } }
I20260620 04:36:06.663815 8214 raft_consensus.cc:1261] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.663815 8354 raft_consensus.cc:1261] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Refusing update from remote peer 16704c72df034068816e50e2ea2bf41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:06.664249 8439 consensus_queue.cc:1035] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc964df78c54e4d89103eb0f5512fe3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.664441 8448 consensus_queue.cc:1035] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142800ec9a84ef99e6687223beb13d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:06.694550 7915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.694723 7915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:06.696700 7915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 195e0f7463e54e7cb8169459bd7372db: Sending DeleteTablet for 3 replicas of tablet ac6d6e01a50746dc8037b9394284ea33
I20260620 04:36:06.697238 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6d6e01a50746dc8037b9394284ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51856
I20260620 04:36:06.697270 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6d6e01a50746dc8037b9394284ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48234
I20260620 04:36:06.697422 8454 tablet_replica.cc:330] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: stopping tablet replica
I20260620 04:36:06.697386 8333 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6d6e01a50746dc8037b9394284ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260620 04:36:06.697546 8454 raft_consensus.cc:2229] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.697579 8453 tablet_replica.cc:330] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: stopping tablet replica
I20260620 04:36:06.697577 8452 tablet_replica.cc:330] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f: stopping tablet replica
I20260620 04:36:06.697666 8454 raft_consensus.cc:2258] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.697681 8452 raft_consensus.cc:2229] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:06.697686 8453 raft_consensus.cc:2229] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:06.697811 8453 raft_consensus.cc:2258] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.697866 8452 raft_consensus.cc:2258] T ac6d6e01a50746dc8037b9394284ea33 P 16704c72df034068816e50e2ea2bf41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:06.698324 8454 ts_tablet_manager.cc:1905] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:06.698371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7989
I20260620 04:36:06.700812 8454 ts_tablet_manager.cc:1918] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:36:06.700903 8454 log.cc:1196] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/ac6d6e01a50746dc8037b9394284ea33
I20260620 04:36:06.701123 8454 ts_tablet_manager.cc:1939] T ac6d6e01a50746dc8037b9394284ea33 P 0dc964df78c54e4d89103eb0f5512fe3: Deleting consensus metadata
I20260620 04:36:06.704015 7902 catalog_manager.cc:4841] TS 0dc964df78c54e4d89103eb0f5512fe3 (127.0.98.2:39717): tablet ac6d6e01a50746dc8037b9394284ea33 (table test-table [id=e1523b5347c4423f85525500c6299e1f]) successfully deleted
I20260620 04:36:06.704398 8453 ts_tablet_manager.cc:1905] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:36:06.706408 7902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43585 (error 108)
I20260620 04:36:06.706564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8129
I20260620 04:36:06.706612 8453 ts_tablet_manager.cc:1918] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:36:06.706709 8453 log.cc:1196] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/ac6d6e01a50746dc8037b9394284ea33
I20260620 04:36:06.706943 8453 ts_tablet_manager.cc:1939] T ac6d6e01a50746dc8037b9394284ea33 P 0142800ec9a84ef99e6687223beb13d4: Deleting consensus metadata
I20260620 04:36:06.707904 7901 catalog_manager.cc:4841] TS 0142800ec9a84ef99e6687223beb13d4 (127.0.98.3:33119): tablet ac6d6e01a50746dc8037b9394284ea33 (table test-table [id=e1523b5347c4423f85525500c6299e1f]) successfully deleted
I20260620 04:36:06.713394 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:06.713028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.0.1:51888)
0620 04:36:06.713237 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:06.713313 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
W20260620 04:36:06.713645 7902 catalog_manager.cc:4584] TS 16704c72df034068816e50e2ea2bf41f (127.0.98.1:43585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac6d6e01a50746dc8037b9394284ea33: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43585: connect: Connection refused (error 111)
I20260620 04:36:06.714877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8269
I20260620 04:36:06.720968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 7885
2026-06-20T04:36:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3386 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[8510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[8510](info): set up 2 sockets
Jun 20 04:36:06 dist-test-slave-zmf4 krb5kdc[8510](info): commencing operation
krb5kdc: starting...
W20260620 04:36:07.804899 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:36:07 dist-test-slave-zmf4 krb5kdc[8510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:36:07.931119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:08.045351 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:08.045629 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:08.045675 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:08.049873 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:08.049942 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:08.049962 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:08.049979 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:08.049995 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:08.056288 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:08.057646 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:08.058820 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:08.067149 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:08.067138 8529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:08.067138 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:08.067363 8523 server_base.cc:1034] running on GCE node
I20260620 04:36:08.068122 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:08.069371 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:08.070750 8523 hybrid_clock.cc:648] HybridClock initialized: now 1781930168070723 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T04:36:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1357 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:36:08 dist-test-slave-zmf4 krb5kdc[8536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:36:08 dist-test-slave-zmf4 krb5kdc[8536](info): set up 2 sockets
Jun 20 04:36:08 dist-test-slave-zmf4 krb5kdc[8536](info): commencing operation
krb5kdc: starting...
W20260620 04:36:09.145540 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:36:09.271126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:09.386569 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:09.386883 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:09.386965 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:09.391069 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:09.391162 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:09.391209 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:09.391263 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:09.391299 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:09.396934 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:09.398038 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:09.399040 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:09.407444 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:09.407440 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:09.407485 8549 server_base.cc:1034] running on GCE node
W20260620 04:36:09.407421 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:09.408182 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:09.409197 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:09.410362 8549 hybrid_clock.cc:648] HybridClock initialized: now 1781930169410337 us; error 41 us; skew 500 ppm
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:09.417449 8549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:36:09.418311 8549 webserver.cc:460] Webserver started at http://127.0.98.62:35527/ using document root <none> and password file <none>
I20260620 04:36:09.418876 8549 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:09.418962 8549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:09.419185 8549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:09.421252 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "9384fe41e5234d8c89e11e5f04d4af23"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "567961075d6d6dc68fe470e7a1576ef6"
server_key_iv: "b2ebaad11f183bc3dbbe92b3b1c2a788"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.421813 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "9384fe41e5234d8c89e11e5f04d4af23"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "567961075d6d6dc68fe470e7a1576ef6"
server_key_iv: "b2ebaad11f183bc3dbbe92b3b1c2a788"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.425504 8549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:09.427845 8564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.429090 8549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:09.429219 8549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "9384fe41e5234d8c89e11e5f04d4af23"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "567961075d6d6dc68fe470e7a1576ef6"
server_key_iv: "b2ebaad11f183bc3dbbe92b3b1c2a788"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.429322 8549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:09.450176 8549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:09.453804 8549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:09.454205 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:09.461356 8549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37569
I20260620 04:36:09.461419 8616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37569 every 8 connection(s)
I20260620 04:36:09.462301 8549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:09.465219 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:09.468035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8549
I20260620 04:36:09.468133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:36:09.468442 392 external_mini_cluster.cc:1428] Setting key 7c534b2d774747eca5ce5acd8b7d44dc
I20260620 04:36:09.471690 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Bootstrap starting.
I20260620 04:36:09.474375 8617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:09.475167 8617 log.cc:824] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:09.477370 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: No bootstrap required, opened a new log
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:09.481225 8617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:09.481508 8617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:09.481556 8617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9384fe41e5234d8c89e11e5f04d4af23, State: Initialized, Role: FOLLOWER
I20260620 04:36:09.482040 8617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:09.482174 8617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:09.482216 8617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:09.482276 8617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:09.483317 8617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:09.483690 8617 leader_election.cc:302] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9384fe41e5234d8c89e11e5f04d4af23; no voters:
I20260620 04:36:09.484097 8617 leader_election.cc:288] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:36:09.484165 8622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:09.484409 8622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 LEADER]: Becoming Leader. State: Replica: 9384fe41e5234d8c89e11e5f04d4af23, State: Running, Role: LEADER
I20260620 04:36:09.484784 8622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:09.484915 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:09.469920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.62:37569)
0620 04:36:09.470364 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:09.470385 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:09.470430 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:36:09.471282 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:09.471304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:09.471747 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:09.472134 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:09.473190 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.474684 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.475760 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.476398 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.476541 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:09.479191 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:09.479197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:09.479213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:09.479242 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:09.481834 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:09.482387 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:09.482393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:09.482403 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:09.482512 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:09.482790 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:09.482793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:09.482795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:09.483165 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:09.483366 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:09.483630 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 04:36:09.483954 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260620 04:36:09.485471 8617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:09.486063 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9384fe41e5234d8c89e11e5f04d4af23. Latest consensus state: current_term: 1 leader_uuid: "9384fe41e5234d8c89e11e5f04d4af23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } } }
I20260620 04:36:09.486187 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:09.486925 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:09.487350 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9384fe41e5234d8c89e11e5f04d4af23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } } }
I20260620 04:36:09.487519 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:09.490389 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:09.496007 8627 catalog_manager.cc:1347] Generated new cluster ID: f75701831dee4f10bf58803c0865c058
I20260620 04:36:09.496074 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:09.521288 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:36:09.522264 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:09.538920 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Generated new TSK 0
I20260620 04:36:09.539659 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:36:09.577733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:09.689484 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:09.689787 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:09.689859 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:09.694098 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:09.694198 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:09.694289 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:09.699618 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:09.700734 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:09.701767 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:09.711838 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:09.711845 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:09.711777 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:09.712118 8643 server_base.cc:1034] running on GCE node
I20260620 04:36:09.712544 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:09.713100 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:09.714265 8643 hybrid_clock.cc:648] HybridClock initialized: now 1781930169714248 us; error 28 us; skew 500 ppm
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:09.717054 8643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:36:09.717900 8643 webserver.cc:460] Webserver started at http://127.0.98.1:39583/ using document root <none> and password file <none>
I20260620 04:36:09.718452 8643 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:09.718539 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:09.718755 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:09.720780 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "f78f187db7de42a7b20ceb5b63e50e5e"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "ffb495d57e0895af82b9c6518cfe4f31"
server_key_iv: "a66e9c270583bd51f085433846fb303a"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.721334 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f78f187db7de42a7b20ceb5b63e50e5e"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "ffb495d57e0895af82b9c6518cfe4f31"
server_key_iv: "a66e9c270583bd51f085433846fb303a"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.724964 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:36:09.727339 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.728571 8643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:09.728700 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "f78f187db7de42a7b20ceb5b63e50e5e"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "ffb495d57e0895af82b9c6518cfe4f31"
server_key_iv: "a66e9c270583bd51f085433846fb303a"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.728803 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:09.752564 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:09.755756 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:09.755960 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:09.756564 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:09.757362 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:09.757427 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.757493 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:09.757539 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.766460 8643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37215
I20260620 04:36:09.766485 8771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37215 every 8 connection(s)
I20260620 04:36:09.767331 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:09.774690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8643
I20260620 04:36:09.774869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:36:09.775152 392 external_mini_cluster.cc:1428] Setting key d59ebfff5422bf85a893ec7ba6d4651b
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:09.781958 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:09.769237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52509 (local address 127.0.98.62:37569)
0620 04:36:09.769374 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:09.769377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:09.770196 (+ 819us) server_negotiation.cc:407] Connection header received
0620 04:36:09.771559 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:09.771563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:09.771629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:09.771726 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:09.772851 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.773383 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.774735 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.775049 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.775173 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:09.777948 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:09.777953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:09.777956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:09.777982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:09.779749 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:09.780322 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:09.780329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:09.780334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:09.780441 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:09.780802 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:09.780806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:09.780809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:09.781003 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:09.781114 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:09.781641 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 04:36:09.781773 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:36:09.782737 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:09.769577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:52509)
0620 04:36:09.770036 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:09.770077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:09.771354 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:09.771890 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:09.771896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:09.772264 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:09.772711 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:09.772723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.773668 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.773672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:09.774614 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:09.774624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.775208 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.775212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:09.775378 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:09.776084 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:09.776101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:09.777777 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:09.779868 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:09.779875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:09.779890 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:09.780171 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:09.780548 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:09.780550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:09.780551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:09.780688 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:09.781137 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:09.781144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:09.781384 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:36:09.781572 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 04:36:09.781909 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260620 04:36:09.784293 8772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37569
I20260620 04:36:09.784528 8772 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:09.785027 8772 heartbeater.cc:511] Master 127.0.98.62:37569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:36:09.786600 8581 ts_manager.cc:194] Registered new tserver with Master: f78f187db7de42a7b20ceb5b63e50e5e (127.0.98.1:37215)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:36:09.787935 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:36:09.802726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:09.918699 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:09.918954 8777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:09.918998 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:09.923117 8777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:09.923190 8777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:09.923259 8777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:09.929149 8777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:09.930143 8777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:09.931314 8777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:09.941215 8782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:09.941206 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:09.941381 8777 server_base.cc:1034] running on GCE node
W20260620 04:36:09.941203 8783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:09.941877 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:09.942467 8777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:09.943624 8777 hybrid_clock.cc:648] HybridClock initialized: now 1781930169943610 us; error 25 us; skew 500 ppm
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:09.946382 8777 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:36:09.947206 8777 webserver.cc:460] Webserver started at http://127.0.98.2:37667/ using document root <none> and password file <none>
I20260620 04:36:09.947754 8777 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:09.947841 8777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:09.948058 8777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:09.950057 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "becbbae724cb478bb8d25eb59f3dd72c"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "9b621298f561c50ed6731b56ae75ae87"
server_key_iv: "5f33c4f4c01f57e192e2d50413638c8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.950603 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "becbbae724cb478bb8d25eb59f3dd72c"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "9b621298f561c50ed6731b56ae75ae87"
server_key_iv: "5f33c4f4c01f57e192e2d50413638c8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.954193 8777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:36:09.956640 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.958030 8777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:09.958180 8777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "becbbae724cb478bb8d25eb59f3dd72c"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "9b621298f561c50ed6731b56ae75ae87"
server_key_iv: "5f33c4f4c01f57e192e2d50413638c8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:09.958300 8777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:09.976387 8777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:09.979566 8777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:09.979764 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:09.980356 8777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:09.981133 8777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:09.981197 8777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.981261 8777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:09.981307 8777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:09.990897 8777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45121
I20260620 04:36:09.990916 8905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45121 every 8 connection(s)
I20260620 04:36:09.991760 8777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:09.999231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8777
I20260620 04:36:09.999383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:36:09.999646 392 external_mini_cluster.cc:1428] Setting key b14838b2df4bef24fc59317c845f84ad
Jun 20 04:36:09 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:10.006354 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:09.993494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50443 (local address 127.0.98.62:37569)
0620 04:36:09.993656 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:09.993660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:09.994398 (+ 738us) server_negotiation.cc:407] Connection header received
0620 04:36:09.995764 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:09.995768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:09.995821 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:09.995906 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:09.997187 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.997721 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.998837 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.999181 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.999320 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.001960 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:10.001964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:10.001967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:10.001991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:10.003815 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:10.004495 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:10.004501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:10.004503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:10.004596 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:10.005032 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:10.005035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:10.005036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:10.005284 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:10.005412 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:10.006100 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 04:36:10.006225 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:36:10.007577 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:09.993846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:50443)
0620 04:36:09.994250 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:09.994288 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:09.995561 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:09.996074 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:09.996080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:09.996572 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:09.997034 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:09.997047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.997846 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.997859 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:09.998712 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:09.998720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:09.999358 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:09.999362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:09.999574 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.000256 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.000277 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.001803 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:10.003942 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:10.003951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:10.003969 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:10.004369 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:10.004697 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:10.004703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:10.004708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:10.004892 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:10.005421 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:10.005431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:10.005780 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 04:36:10.005995 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 04:36:10.006378 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260620 04:36:10.008886 8906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37569
I20260620 04:36:10.009142 8906 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:10.009624 8906 heartbeater.cc:511] Master 127.0.98.62:37569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:36:10.010735 8581 ts_manager.cc:194] Registered new tserver with Master: becbbae724cb478bb8d25eb59f3dd72c (127.0.98.2:45121)
I20260620 04:36:10.011339 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:36:10.028535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:10.141073 8911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:10.141330 8911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:10.141373 8911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:10.145594 8911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:10.145649 8911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:10.145730 8911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:10.150967 8911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:10.151976 8911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:10.153013 8911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:10.162465 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.162551 8911 server_base.cc:1034] running on GCE node
W20260620 04:36:10.162473 8919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:10.162472 8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.163161 8911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:10.163791 8911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:10.164961 8911 hybrid_clock.cc:648] HybridClock initialized: now 1781930170164930 us; error 41 us; skew 500 ppm
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:10.167843 8911 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:36:10.168727 8911 webserver.cc:460] Webserver started at http://127.0.98.3:46547/ using document root <none> and password file <none>
I20260620 04:36:10.169266 8911 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:10.169332 8911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:10.169535 8911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:10.171577 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "31952c8249004dd09de734c237584ece"
format_stamp: "Formatted at 2026-06-20 04:36:10 on dist-test-slave-zmf4"
server_key: "ba64574b66f3a6cce9a3c5551776f489"
server_key_iv: "a7be7f57e9a8b27306f9427659a13045"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.172081 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31952c8249004dd09de734c237584ece"
format_stamp: "Formatted at 2026-06-20 04:36:10 on dist-test-slave-zmf4"
server_key: "ba64574b66f3a6cce9a3c5551776f489"
server_key_iv: "a7be7f57e9a8b27306f9427659a13045"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.175738 8911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:10.178563 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.179917 8911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:10.180018 8911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "31952c8249004dd09de734c237584ece"
format_stamp: "Formatted at 2026-06-20 04:36:10 on dist-test-slave-zmf4"
server_key: "ba64574b66f3a6cce9a3c5551776f489"
server_key_iv: "a7be7f57e9a8b27306f9427659a13045"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.180182 8911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:10.205839 8911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:10.209180 8911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:10.209489 8911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:10.210107 8911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:10.210923 8911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:10.210970 8911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.211035 8911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:10.211068 8911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.220497 8911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45491
I20260620 04:36:10.220620 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45491 every 8 connection(s)
I20260620 04:36:10.221570 8911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:10.224934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 8911
I20260620 04:36:10.225062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:10.225332 392 external_mini_cluster.cc:1428] Setting key 904e7d614cd98ce6c389ef7f3d5cdea3
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:10.235556 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.223362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53083 (local address 127.0.98.62:37569)
0620 04:36:10.223500 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.223504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.224365 (+ 861us) server_negotiation.cc:407] Connection header received
0620 04:36:10.225797 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.225802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.225854 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.225931 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.227331 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.227867 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.229089 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.229321 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.229417 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.231846 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:10.231850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:10.231852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:10.231877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:10.233619 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:10.234107 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:10.234111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:10.234116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:10.234197 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:10.234527 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:10.234531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:10.234532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:10.234714 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:10.234816 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:10.235274 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 04:36:10.235388 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:36:10.236369 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.223773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:53083)
0620 04:36:10.224185 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.224273 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.225603 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.226231 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.226240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.226652 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.227181 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.227203 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.228008 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.228011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.228958 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.228967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.229484 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.229487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.229628 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.230198 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.230228 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.231677 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:10.233735 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:10.233741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:10.233754 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:10.234004 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:10.234289 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:10.234292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:10.234294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:10.234426 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:10.234813 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:10.234820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:10.235048 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 04:36:10.235236 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 04:36:10.235530 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260620 04:36:10.237581 9040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37569
I20260620 04:36:10.237821 9040 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:10.238363 9040 heartbeater.cc:511] Master 127.0.98.62:37569 requested a full tablet report, sending...
I20260620 04:36:10.239378 8581 ts_manager.cc:194] Registered new tserver with Master: 31952c8249004dd09de734c237584ece (127.0.98.3:45491)
I20260620 04:36:10.239981 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53083
I20260620 04:36:10.251358 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:36:10.327556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8643
I20260620 04:36:10.333734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8777
I20260620 04:36:10.339066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8911
I20260620 04:36:10.344487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 8549
I20260620 04:36:10.350772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37569
--webserver_interface=127.0.98.62
--webserver_port=35527
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:10.466241 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:10.466545 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:10.466619 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:10.474864 9049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:10.474968 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:10.475011 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:10.475042 9049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:10.475075 9049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:10.481603 9049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:10.482726 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:10.483731 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:10.492555 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.492718 9049 server_base.cc:1034] running on GCE node
W20260620 04:36:10.492548 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:10.492548 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.493429 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:10.494524 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:10.495699 9049 hybrid_clock.cc:648] HybridClock initialized: now 1781930170495681 us; error 33 us; skew 500 ppm
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:10.498559 9049 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:36:10.499575 9049 webserver.cc:460] Webserver started at http://127.0.98.62:35527/ using document root <none> and password file <none>
I20260620 04:36:10.500144 9049 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:10.500288 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:10.504076 9049 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:36:10.506246 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.507547 9049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:36:10.507716 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "9384fe41e5234d8c89e11e5f04d4af23"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "567961075d6d6dc68fe470e7a1576ef6"
server_key_iv: "b2ebaad11f183bc3dbbe92b3b1c2a788"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.508303 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:10.524446 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:10.527505 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:10.527729 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:10.535122 9049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37569
I20260620 04:36:10.535136 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37569 every 8 connection(s)
I20260620 04:36:10.536237 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:10.537434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9049
I20260620 04:36:10.537856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37215
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37569
--webserver_port=39583
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:36:10.540472 9117 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:36:10.542119 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Bootstrap starting.
I20260620 04:36:10.558594 9117 log.cc:824] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:10.565055 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:36:10.565410 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Bootstrap complete.
I20260620 04:36:10.569543 9117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:10.569948 9117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9384fe41e5234d8c89e11e5f04d4af23, State: Initialized, Role: FOLLOWER
I20260620 04:36:10.570597 9117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:10.570803 9117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:10.570886 9117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:10.570994 9117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:36:10.573173 9117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:10.573514 9117 leader_election.cc:302] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9384fe41e5234d8c89e11e5f04d4af23; no voters:
I20260620 04:36:10.573872 9117 leader_election.cc:288] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:36:10.573998 9121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:36:10.574226 9121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [term 2 LEADER]: Becoming Leader. State: Replica: 9384fe41e5234d8c89e11e5f04d4af23, State: Running, Role: LEADER
I20260620 04:36:10.574592 9121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } }
I20260620 04:36:10.575155 9117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:10.576107 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9384fe41e5234d8c89e11e5f04d4af23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } } }
I20260620 04:36:10.576439 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:10.576061 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9384fe41e5234d8c89e11e5f04d4af23. Latest consensus state: current_term: 2 leader_uuid: "9384fe41e5234d8c89e11e5f04d4af23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9384fe41e5234d8c89e11e5f04d4af23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37569 } } }
I20260620 04:36:10.576779 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:10.577098 9130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:10.580402 9130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:10.581067 9130 catalog_manager.cc:1259] Loaded cluster ID: f75701831dee4f10bf58803c0865c058
I20260620 04:36:10.581249 9130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:10.583688 9130 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:10.584589 9130 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9384fe41e5234d8c89e11e5f04d4af23: Loaded TSK: 0
I20260620 04:36:10.585867 9130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:36:10.657320 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:10.657788 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:10.657905 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:10.662320 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:10.662405 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:10.662496 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:10.667694 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39583
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:10.668751 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:10.669721 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:10.680239 9144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:10.680266 9146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:10.680248 9143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.680830 9119 server_base.cc:1034] running on GCE node
I20260620 04:36:10.681257 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:10.681913 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:10.683137 9119 hybrid_clock.cc:648] HybridClock initialized: now 1781930170683096 us; error 48 us; skew 500 ppm
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930170, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:10.686126 9119 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:36:10.686968 9119 webserver.cc:460] Webserver started at http://127.0.98.1:39583/ using document root <none> and password file <none>
I20260620 04:36:10.687525 9119 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:10.687640 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:10.691711 9119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:10.693976 9153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.695261 9119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:36:10.695410 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "f78f187db7de42a7b20ceb5b63e50e5e"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "ffb495d57e0895af82b9c6518cfe4f31"
server_key_iv: "a66e9c270583bd51f085433846fb303a"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.695950 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:10.722121 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:10.725503 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:10.725741 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:10.726382 9119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:10.727244 9119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:10.727320 9119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.727397 9119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:10.727454 9119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.738619 9119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37215
I20260620 04:36:10.738684 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37215 every 8 connection(s)
I20260620 04:36:10.739629 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:10.744521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9119
I20260620 04:36:10.744956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45121
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37569
--webserver_port=37667
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:36:10.752938 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.741763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51299 (local address 127.0.98.62:37569)
0620 04:36:10.742201 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.742220 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.742471 (+ 251us) server_negotiation.cc:407] Connection header received
0620 04:36:10.743944 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.743955 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.744304 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.744648 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.745827 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.746786 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.747884 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.748446 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.748624 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.752178 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51299: BlockingRecv error: recv got EOF from 127.0.98.1:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260620 04:36:10.752948 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.741865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:51299)
0620 04:36:10.742321 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.742362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.743551 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.744714 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.744721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.745219 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.745663 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.745677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.746923 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.746926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.747762 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.747771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.748612 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.748616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.748793 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.749384 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.749399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.751992 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":169,"threads_started":1}
W20260620 04:36:10.753544 9267 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37569 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37569: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:10.758940 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.753993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:53321)
0620 04:36:10.754107 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.754121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.754200 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.754422 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.754424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.754581 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.754662 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.754667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.755370 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.755375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.755902 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.755909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.756406 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.756409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.756504 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.757013 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.757024 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.758801 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 04:36:10.759013 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.754091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53321 (local address 127.0.98.62:37569)
0620 04:36:10.754215 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.754221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.754238 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:36:10.754277 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.754279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.754325 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.754408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.754769 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.755265 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.756025 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.756301 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.756403 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.758909 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53321: BlockingRecv error: recv got EOF from 127.0.98.1:53321 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:10.764384 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.759463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:48365)
0620 04:36:10.759555 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.759569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.759640 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.759848 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.759850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.759998 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.760061 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.760066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.760781 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.760786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.761282 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.761289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.761750 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.761753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.761834 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.762311 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.762322 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.764251 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 04:36:10.764660 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.759543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48365 (local address 127.0.98.62:37569)
0620 04:36:10.759641 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.759645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.759658 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:10.759710 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.759713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.759748 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.759835 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.760166 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.760675 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.761393 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.761629 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.761744 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.764417 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48365: BlockingRecv error: recv got EOF from 127.0.98.1:48365 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 04:36:10.764778 9267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:36:10.857946 9271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:10.858201 9271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:10.858243 9271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:10.862458 9271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:10.862524 9271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:10.862586 9271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:10.867884 9271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37667
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:10.868911 9271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:10.869899 9271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:10.879581 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:10.879582 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.879683 9271 server_base.cc:1034] running on GCE node
W20260620 04:36:10.879611 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:10.880268 9271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:10.880882 9271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:10.882077 9271 hybrid_clock.cc:648] HybridClock initialized: now 1781930170882047 us; error 45 us; skew 500 ppm
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930170, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:10.884948 9271 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:36:10.885824 9271 webserver.cc:460] Webserver started at http://127.0.98.2:37667/ using document root <none> and password file <none>
I20260620 04:36:10.886368 9271 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:10.886485 9271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:10.890388 9271 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:10.892535 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.893993 9271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:36:10.894135 9271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "becbbae724cb478bb8d25eb59f3dd72c"
format_stamp: "Formatted at 2026-06-20 04:36:09 on dist-test-slave-zmf4"
server_key: "9b621298f561c50ed6731b56ae75ae87"
server_key_iv: "5f33c4f4c01f57e192e2d50413638c8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:10.894675 9271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:10.943171 9271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:10.946808 9271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:10.947041 9271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:10.947662 9271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:10.948499 9271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:10.948570 9271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.948637 9271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:10.948680 9271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:10.958822 9271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45121
I20260620 04:36:10.958842 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45121 every 8 connection(s)
I20260620 04:36:10.959859 9271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:10.961971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9271
I20260620 04:36:10.962391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45491
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37569
--webserver_port=46547
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:36:10.972275 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.961756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33387 (local address 127.0.98.62:37569)
0620 04:36:10.961915 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.961920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.962769 (+ 849us) server_negotiation.cc:407] Connection header received
0620 04:36:10.964523 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.964527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.964598 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.964701 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.965822 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.966694 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.968009 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.968443 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.968585 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.972060 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33387: BlockingRecv error: recv got EOF from 127.0.98.2:33387 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:36:10.972872 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.962132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:33387)
0620 04:36:10.962590 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.962649 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.964318 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.964872 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.964879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.965225 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.965680 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.965693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.966837 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.966840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.967872 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.967882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.968600 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.968603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.968755 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.969401 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.969417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.971853 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":136,"threads_started":1}
W20260620 04:36:10.973778 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37569 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37569: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:10.981547 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.974253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:40643)
0620 04:36:10.974611 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.974629 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.974759 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.975112 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.975115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.975287 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.975399 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.975405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.976549 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.976552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.977524 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.977535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.977970 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.977974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.978066 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.978578 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.978592 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.981366 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200}
W20260620 04:36:10.981652 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.974417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40643 (local address 127.0.98.62:37569)
0620 04:36:10.974592 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.974599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.974679 (+ 80us) server_negotiation.cc:407] Connection header received
0620 04:36:10.974861 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.974864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.974911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.975001 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.975529 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.976413 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.977628 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.977855 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.977944 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.981525 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40643: BlockingRecv error: recv got EOF from 127.0.98.2:40643 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:10 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:10.987943 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:10.982166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:36145)
0620 04:36:10.982431 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:10.982452 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:10.982556 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:10.982805 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:10.982808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:10.982967 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:10.983037 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.983042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.983767 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.983772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.984360 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:10.984368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.984795 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.984798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:10.984876 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.985505 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:10.985524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:10.987816 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194}
W20260620 04:36:10.988056 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:10.982293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36145 (local address 127.0.98.62:37569)
0620 04:36:10.982404 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:10.982408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:10.982514 (+ 106us) server_negotiation.cc:407] Connection header received
0620 04:36:10.982657 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:10.982660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:10.982706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:10.982791 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:10.983145 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.983658 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.984471 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:10.984686 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:10.984793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:10.987931 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36145: BlockingRecv error: recv got EOF from 127.0.98.2:36145 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 04:36:10.988171 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:36:11.084069 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:11.084468 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:11.084538 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:11.088713 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:11.088768 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:11.088829 9403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:11.093994 9403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46547
--tserver_master_addrs=127.0.98.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:11.095088 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:11.096041 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:11.105686 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:11.105799 9403 server_base.cc:1034] running on GCE node
W20260620 04:36:11.105736 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:11.105778 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:11.106359 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:11.106977 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:11.108263 9403 hybrid_clock.cc:648] HybridClock initialized: now 1781930171108235 us; error 31 us; skew 500 ppm
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930171, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:11.110997 9403 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:36:11.111897 9403 webserver.cc:460] Webserver started at http://127.0.98.3:46547/ using document root <none> and password file <none>
I20260620 04:36:11.112605 9403 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:11.112741 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:11.116988 9403 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:36:11.119314 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:11.120692 9403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:11.120837 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "31952c8249004dd09de734c237584ece"
format_stamp: "Formatted at 2026-06-20 04:36:10 on dist-test-slave-zmf4"
server_key: "ba64574b66f3a6cce9a3c5551776f489"
server_key_iv: "a7be7f57e9a8b27306f9427659a13045"
server_key_version: "encryptionkey@0"
I20260620 04:36:11.121372 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:11.155397 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:11.158834 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:11.159052 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:11.159636 9403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:11.160470 9403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:11.160542 9403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:11.160610 9403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:11.160655 9403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:11.169965 9403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45491
I20260620 04:36:11.169981 9531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45491 every 8 connection(s)
I20260620 04:36:11.170940 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:11.180770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9403
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:36:11.182627 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:11.173098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33157 (local address 127.0.98.62:37569)
0620 04:36:11.173260 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.173270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.173941 (+ 671us) server_negotiation.cc:407] Connection header received
0620 04:36:11.175313 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.175317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.175374 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.175480 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.176714 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.177310 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.178527 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.178746 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.178853 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.182508 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33157: BlockingRecv error: recv got EOF from 127.0.98.3:33157 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 04:36:11.183252 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.173304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:33157)
0620 04:36:11.173776 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:11.173816 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:11.175056 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:11.175649 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:11.175655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:11.176005 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:11.176580 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.176594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.177425 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.177428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.178397 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.178406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.178889 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.178894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.179054 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.179594 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:11.179610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:11.182401 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
W20260620 04:36:11.183755 9532 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37569 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37569: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:11.190013 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.184210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:51923)
0620 04:36:11.184362 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:11.184375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:11.184445 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:11.184721 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:11.184724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:11.184887 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:11.185007 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.185012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.186048 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.186051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.186975 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.186987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.187557 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.187561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.187674 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.188118 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:11.188137 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:11.189926 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 04:36:11.190218 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:11.184273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51923 (local address 127.0.98.62:37569)
0620 04:36:11.184380 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.184384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.184406 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:36:11.184569 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.184571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.184609 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.184696 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.185113 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.185781 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.187092 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.187435 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.187537 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.190083 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51923: BlockingRecv error: recv got EOF from 127.0.98.3:51923 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:36:11.192036 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.181816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.0.98.62:37569)
0620 04:36:11.182101 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.182104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.182116 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:36:11.182146 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.182148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.182184 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.182304 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.183089 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.183753 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.184969 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.185293 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.185400 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.187095 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:11.187102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:11.187119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:11.187150 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:11.189143 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:11.189873 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:11.189879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:11.189893 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:11.190079 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:11.190758 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:11.190762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:11.190764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:11.191205 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:11.191429 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:11.191643 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 04:36:11.191875 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:11.196128 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.190543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:56807)
0620 04:36:11.190648 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:11.190670 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:11.190726 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:11.190999 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:11.191001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:11.191113 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:11.191175 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.191179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.192161 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.192166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.192713 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.192722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.193290 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.193294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.193376 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.194046 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:11.194063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:11.195987 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 04:36:11.196228 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:11.190652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56807 (local address 127.0.98.62:37569)
0620 04:36:11.190799 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.190805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.190819 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:11.190851 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.190854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.190891 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.190973 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.191276 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.192044 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.192823 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.193154 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.193241 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.196090 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56807: BlockingRecv error: recv got EOF from 127.0.98.3:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 04:36:11.196364 9532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:11.771623 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.765413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:48843)
0620 04:36:11.765723 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:11.765738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:11.765833 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:11.766125 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:11.766128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:11.766313 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:11.766418 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.766423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.767228 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.767231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.768023 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.768033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.768686 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.768690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.768795 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.769318 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:11.769330 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:11.771488 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
W20260620 04:36:11.771791 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:11.765558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48843 (local address 127.0.98.62:37569)
0620 04:36:11.765787 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.765791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.765805 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:11.765956 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.765959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.766011 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.766092 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.766541 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.767084 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.768183 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.768515 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.768632 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.771650 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48843: BlockingRecv error: recv got EOF from 127.0.98.1:48843 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:11 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:11.994865 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:11.988854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:34475)
0620 04:36:11.989112 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:11.989134 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:11.989228 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:11.989561 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:11.989564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:11.989747 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:11.989862 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.989868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.990804 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.990813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.991616 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:11.991626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.992158 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.992160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:11.992288 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.992855 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:11.992867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:11.994683 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260620 04:36:11.994921 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:11.988994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34475 (local address 127.0.98.62:37569)
0620 04:36:11.989143 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:11.989147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:11.989192 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:36:11.989312 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:11.989314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:11.989357 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:11.989502 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:11.989998 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.990664 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.991743 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:11.992051 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:11.992138 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:11.994807 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34475: BlockingRecv error: recv got EOF from 127.0.98.2:34475 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:12.203233 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:12.197074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:51887)
0620 04:36:12.197341 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:12.197354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:12.197481 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:12.197754 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:12.197757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:12.198015 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:12.198137 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.198143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.199032 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.199036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.199803 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.199813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.200337 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.200341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.200416 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:12.200936 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:12.200949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:12.203056 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
W20260620 04:36:12.203500 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:12.197127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51887 (local address 127.0.98.62:37569)
0620 04:36:12.197285 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:12.197292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:12.197496 (+ 204us) server_negotiation.cc:407] Connection header received
0620 04:36:12.197569 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:12.197572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:12.197620 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:12.197684 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:12.198286 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.198810 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.199934 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.200215 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.200302 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:12.203371 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51887: BlockingRecv error: recv got EOF from 127.0.98.3:51887 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:12.779096 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:12.772617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:44829)
0620 04:36:12.772878 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:12.772910 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:12.773023 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:12.773432 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:12.773435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:12.773725 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:12.773836 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.773842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.774642 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.774648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.775204 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.775212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.775716 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.775721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.775804 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:12.776620 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:12.776637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:12.778885 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
W20260620 04:36:12.779134 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:12.772673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44829 (local address 127.0.98.62:37569)
0620 04:36:12.772954 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:12.772965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:12.772977 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:36:12.773209 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:12.773213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:12.773278 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:12.773408 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:12.773994 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.774532 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.775325 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.775582 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.775692 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:12.779015 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44829: BlockingRecv error: recv got EOF from 127.0.98.1:44829 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":57,"threads_started":1}
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:12 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:13.002261 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:12.995973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:58935)
0620 04:36:12.996406 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:12.996431 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:12.996547 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:12.996896 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:12.996898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:12.997096 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:12.997202 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.997207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.998188 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.998192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.998896 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:12.998925 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.999549 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.999554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:12.999641 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.000294 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:13.000305 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:13.002127 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
W20260620 04:36:13.002419 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:12.996137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58935 (local address 127.0.98.62:37569)
0620 04:36:12.996320 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:12.996324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:12.996543 (+ 219us) server_negotiation.cc:407] Connection header received
0620 04:36:12.996630 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:12.996632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:12.996691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:12.996771 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:12.997509 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.998045 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.999148 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:12.999365 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:12.999460 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.002292 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58935: BlockingRecv error: recv got EOF from 127.0.98.2:58935 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 04:36:13 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:13 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:13.209744 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:13.204056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:41903)
0620 04:36:13.204351 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:13.204364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:13.204457 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:13.204765 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:13.204768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:13.204942 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:13.205039 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:13.205052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.205839 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.205842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:13.206407 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:13.206415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.206940 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.206943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:13.207012 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.207532 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:13.207544 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:13.209536 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260620 04:36:13.209853 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:13.204234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41903 (local address 127.0.98.62:37569)
0620 04:36:13.204358 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:13.204362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:13.204410 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:36:13.204544 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:13.204546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:13.204601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:13.204689 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:13.205191 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.205714 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.206559 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.206792 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.206881 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.209714 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41903: BlockingRecv error: recv got EOF from 127.0.98.3:41903 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 04:36:13 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:13 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:13.785873 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:13.779991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:44129)
0620 04:36:13.780301 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:13.780314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:13.780404 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:13.780688 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:13.780691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:13.780874 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:13.780968 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:13.780974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.781862 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.781867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:13.782712 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:13.782727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.783289 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.783292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:13.783353 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.783854 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:13.783865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:13.785704 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
W20260620 04:36:13.785933 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:13.780144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44129 (local address 127.0.98.62:37569)
0620 04:36:13.780402 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:13.780405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:13.780420 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:13.780463 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:13.780465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:13.780520 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:13.780608 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:13.781123 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.781705 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.782896 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:13.783167 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:13.783261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:13.785819 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44129: BlockingRecv error: recv got EOF from 127.0.98.1:44129 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:14.008818 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:14.003109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:58723)
0620 04:36:14.003421 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:14.003439 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:14.003542 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:14.003807 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:14.003810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:14.003999 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:14.004108 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.004113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.004995 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.004998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.005578 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.005587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.006079 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.006082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.006173 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.006706 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:14.006718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:14.008687 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
W20260620 04:36:14.008982 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:14.003173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58723 (local address 127.0.98.62:37569)
0620 04:36:14.003329 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:14.003335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:14.003543 (+ 208us) server_negotiation.cc:407] Connection header received
0620 04:36:14.003628 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:14.003630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:14.003679 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:14.003744 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:14.004286 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.004867 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.005713 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.005930 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.006023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.008842 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58723: BlockingRecv error: recv got EOF from 127.0.98.2:58723 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:14.216684 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:14.210740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:43421)
0620 04:36:14.211235 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:14.211249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:14.211388 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:14.211676 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:14.211679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:14.211894 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:14.211992 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.211998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.212824 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.212829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.213407 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.213415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.213872 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.213876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.213968 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.214488 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:14.214500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:14.216553 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":91,"threads_started":1}
W20260620 04:36:14.216797 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:14.210810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43421 (local address 127.0.98.62:37569)
0620 04:36:14.211034 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:14.211039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:14.211327 (+ 288us) server_negotiation.cc:407] Connection header received
0620 04:36:14.211486 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:14.211490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:14.211546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:14.211628 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:14.212109 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.212712 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.213535 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.213762 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.213868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.216690 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43421: BlockingRecv error: recv got EOF from 127.0.98.3:43421 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:14 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:14.795972 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:14.786770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:54825)
0620 04:36:14.787064 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:14.787077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:14.787179 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:14.787434 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:14.787437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:14.787647 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:14.787742 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.787751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.788607 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.788610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.789341 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:14.789350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.789906 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.789909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:14.789989 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.793960 (+ 3971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:14.793971 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:14.795848 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20260620 04:36:14.796330 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:14.786816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54825 (local address 127.0.98.62:37569)
0620 04:36:14.787087 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:14.787091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:14.787141 (+ 50us) server_negotiation.cc:407] Connection header received
0620 04:36:14.787256 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:14.787258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:14.787299 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:14.787374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:14.787873 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.788495 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.789487 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:14.789789 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:14.789902 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:14.795996 (+ 6094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54825: BlockingRecv error: recv got EOF from 127.0.98.1:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:15.016023 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:15.009800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:37961)
0620 04:36:15.010068 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:15.010086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:15.010205 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:15.010441 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:15.010443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:15.010623 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:15.010718 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.010722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.011742 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.011746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.012610 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.012620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.013082 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.013085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.013150 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.013772 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:15.013783 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:15.015883 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
W20260620 04:36:15.016304 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:15.009883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37961 (local address 127.0.98.62:37569)
0620 04:36:15.010043 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:15.010047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:15.010182 (+ 135us) server_negotiation.cc:407] Connection header received
0620 04:36:15.010286 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:15.010289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:15.010335 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:15.010412 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:15.010871 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.011613 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.012725 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.012949 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.013043 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.016154 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37961: BlockingRecv error: recv got EOF from 127.0.98.2:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:15.223538 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:15.217557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:47317)
0620 04:36:15.217872 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:15.217892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:15.218026 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:15.218290 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:15.218293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:15.218493 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:15.218582 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.218587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.219343 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.219349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.220057 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.220068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.220631 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.220635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.220723 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.221234 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:15.221246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:15.223412 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
W20260620 04:36:15.223654 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:15.217677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47317 (local address 127.0.98.62:37569)
0620 04:36:15.217839 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:15.217844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:15.217967 (+ 123us) server_negotiation.cc:407] Connection header received
0620 04:36:15.218122 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:15.218127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:15.218182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:15.218269 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:15.218708 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.219234 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.220240 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.220506 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.220638 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.223540 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47317: BlockingRecv error: recv got EOF from 127.0.98.3:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:15 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:15.802472 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:15.796743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:34805)
0620 04:36:15.797073 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:15.797087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:15.797189 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:15.797417 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:15.797419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:15.797629 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:15.797722 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.797728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.798520 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.798523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.799170 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:15.799180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.799674 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.799677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:15.799742 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.800323 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:15.800335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:15.802289 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":88,"threads_started":1}
W20260620 04:36:15.802532 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:15.796848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34805 (local address 127.0.98.62:37569)
0620 04:36:15.797107 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:15.797110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:15.797162 (+ 52us) server_negotiation.cc:407] Connection header received
0620 04:36:15.797268 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:15.797270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:15.797311 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:15.797388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:15.797827 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.798400 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.799315 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:15.799575 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:15.799659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:15.802407 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34805: BlockingRecv error: recv got EOF from 127.0.98.1:34805 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:16.025842 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:16.016937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:38099)
0620 04:36:16.017226 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:16.017251 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:16.017368 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:16.019438 (+ 2070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:16.019441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:16.019651 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:16.019747 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.019753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.020987 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.020991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.021614 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.021646 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.022411 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.022415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.022540 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.023226 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:16.023245 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:16.025711 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260620 04:36:16.025983 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:16.018978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38099 (local address 127.0.98.62:37569)
0620 04:36:16.019149 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:16.019157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:16.019176 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:36:16.019226 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:16.019231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:16.019287 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:16.019377 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:16.019881 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.020861 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.021753 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.022241 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.022367 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.025857 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38099: BlockingRecv error: recv got EOF from 127.0.98.2:38099 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:16.230201 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:16.224428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:41349)
0620 04:36:16.224796 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:16.224813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:16.224959 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:16.225234 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:16.225237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:16.225445 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:16.225563 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.225571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.226328 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.226334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.226892 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.226899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.227363 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.227367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.227451 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.227968 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:16.227980 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:16.230007 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
W20260620 04:36:16.230233 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:16.224615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41349 (local address 127.0.98.62:37569)
0620 04:36:16.224800 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:16.224807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:16.224899 (+ 92us) server_negotiation.cc:407] Connection header received
0620 04:36:16.225065 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:16.225069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:16.225125 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:16.225209 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:16.225682 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.226217 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.227010 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.227242 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.227349 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.230131 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41349: BlockingRecv error: recv got EOF from 127.0.98.3:41349 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:16 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:16.809230 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:16.803404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:39177)
0620 04:36:16.803673 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:16.803687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:16.803774 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:16.804059 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:16.804062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:16.804270 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:16.804366 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.804371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.805257 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.805260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.805902 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:16.805912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.806375 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.806378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:16.806455 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.806962 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:16.806974 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:16.809033 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
W20260620 04:36:16.809298 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:16.803578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39177 (local address 127.0.98.62:37569)
0620 04:36:16.803813 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:16.803816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:16.803829 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:16.803865 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:16.803867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:16.803909 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:16.803996 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:16.804506 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.805114 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.806024 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:16.806246 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:16.806328 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:16.809167 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39177: BlockingRecv error: recv got EOF from 127.0.98.1:39177 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:17.032341 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:17.026647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:46205)
0620 04:36:17.027003 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:17.027018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:17.027156 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:17.027417 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:17.027420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:17.027634 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:17.027722 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.027728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.028533 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.028538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.029088 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.029096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.029547 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.029550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.029632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.030163 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:17.030175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:17.032137 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":86,"threads_started":1}
W20260620 04:36:17.032418 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:17.026790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46205 (local address 127.0.98.62:37569)
0620 04:36:17.026975 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:17.026980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:17.027105 (+ 125us) server_negotiation.cc:407] Connection header received
0620 04:36:17.027262 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:17.027266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:17.027316 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:17.027398 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:17.027838 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.028429 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.029208 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.029427 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.029532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.032297 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46205: BlockingRecv error: recv got EOF from 127.0.98.2:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:17.237353 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:17.231174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:38427)
0620 04:36:17.231434 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:17.231449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:17.231541 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:17.231914 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:17.231916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:17.232094 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:17.232260 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.232271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.233080 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.233083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.233798 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.233808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.234277 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.234279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.234336 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.234913 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:17.234924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:17.237222 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
W20260620 04:36:17.237551 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:17.231341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38427 (local address 127.0.98.62:37569)
0620 04:36:17.231509 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:17.231512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:17.231526 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:17.231673 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:17.231676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:17.231731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:17.231816 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:17.232401 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.232982 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.233919 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.234177 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.234265 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.237376 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38427: BlockingRecv error: recv got EOF from 127.0.98.3:38427 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:17 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:17.815805 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:17.810069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:33473)
0620 04:36:17.810370 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:17.810383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:17.810469 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:17.810699 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:17.810701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:17.810874 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:17.810964 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.810969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.811884 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.811887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.812625 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:17.812635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.813151 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.813154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:17.813219 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.813816 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:17.813829 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:17.815685 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260620 04:36:17.815963 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:17.810189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33473 (local address 127.0.98.62:37569)
0620 04:36:17.810432 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:17.810435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:17.810448 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:17.810552 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:17.810554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:17.810596 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:17.810673 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:17.811098 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.811762 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.812766 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:17.813039 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:17.813134 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:17.815818 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33473: BlockingRecv error: recv got EOF from 127.0.98.1:33473 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:18.039286 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:18.033210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:43209)
0620 04:36:18.033779 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:18.033798 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:18.033888 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:18.034197 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:18.034199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:18.034381 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:18.034476 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.034481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.035262 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.035265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.035818 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.035825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.036353 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.036356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.036451 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.037037 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:18.037048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:18.039176 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":81,"threads_started":1}
W20260620 04:36:18.039395 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:18.033549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43209 (local address 127.0.98.62:37569)
0620 04:36:18.033739 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:18.033742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:18.033974 (+ 232us) server_negotiation.cc:407] Connection header received
0620 04:36:18.034019 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:18.034021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:18.034067 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:18.034131 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:18.034609 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.035143 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.035959 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.036183 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.036339 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.039286 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43209: BlockingRecv error: recv got EOF from 127.0.98.2:43209 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:18.243903 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:18.238491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:43181)
0620 04:36:18.238791 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:18.238804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:18.238940 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:18.239208 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:18.239210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:18.239414 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:18.239503 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.239508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.240298 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.240303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.240857 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.240865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.241331 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.241334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.241418 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.241896 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:18.241907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:18.243744 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
W20260620 04:36:18.244407 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:18.238573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43181 (local address 127.0.98.62:37569)
0620 04:36:18.238732 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:18.238736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:18.238886 (+ 150us) server_negotiation.cc:407] Connection header received
0620 04:36:18.239039 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:18.239043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:18.239091 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:18.239183 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:18.239624 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.240163 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.240985 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.241218 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.241327 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.244265 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43181: BlockingRecv error: recv got EOF from 127.0.98.3:43181 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:18 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:18.822793 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:18.816772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:44977)
0620 04:36:18.817143 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:18.817160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:18.817250 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:18.817587 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:18.817590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:18.817776 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:18.817871 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.817876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.818703 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.818706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.819379 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:18.819390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.819906 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.819909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:18.819973 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.820655 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:18.820666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:18.822667 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
W20260620 04:36:18.823091 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:18.816970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44977 (local address 127.0.98.62:37569)
0620 04:36:18.817193 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:18.817196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:18.817210 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:18.817390 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:18.817392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:18.817448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:18.817527 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:18.818018 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.818555 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.819517 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:18.819780 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:18.819876 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:18.822818 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44977: BlockingRecv error: recv got EOF from 127.0.98.1:44977 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:19.045634 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:19.040157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:53157)
0620 04:36:19.040511 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:19.040524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:19.040656 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:19.040924 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:19.040927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:19.041140 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:19.041228 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.041233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.042135 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.042140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.042702 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.042710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.043159 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.043163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.043247 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.043759 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:19.043770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:19.045519 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
W20260620 04:36:19.045827 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:19.040268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53157 (local address 127.0.98.62:37569)
0620 04:36:19.040439 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:19.040443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:19.040601 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:36:19.040760 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:19.040765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:19.040817 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:19.040903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:19.041345 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.042007 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.042820 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.043048 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.043158 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.045683 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53157: BlockingRecv error: recv got EOF from 127.0.98.2:53157 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:19.250739 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:19.244811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:47943)
0620 04:36:19.245145 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:19.245162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:19.245303 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:19.245580 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:19.245583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:19.245790 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:19.245878 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.245883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.246787 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.246792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.247633 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.247642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.248061 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.248063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.248119 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.248609 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:19.248621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:19.250619 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
W20260620 04:36:19.250927 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:19.244931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47943 (local address 127.0.98.62:37569)
0620 04:36:19.245102 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:19.245107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:19.245245 (+ 138us) server_negotiation.cc:407] Connection header received
0620 04:36:19.245410 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:19.245414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:19.245470 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:19.245556 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:19.246001 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.246667 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.247733 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.247955 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.248038 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.250759 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47943: BlockingRecv error: recv got EOF from 127.0.98.3:47943 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:19 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:19.834678 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:19.823679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:53985)
0620 04:36:19.823967 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:19.823981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:19.824072 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:19.824375 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:19.824378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:19.824623 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:19.824718 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.824723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.825564 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.825567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.826239 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:19.826248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.826683 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.826686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:19.826769 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.832427 (+ 5658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:19.832440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:19.834502 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
W20260620 04:36:19.834753 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:19.823799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53985 (local address 127.0.98.62:37569)
0620 04:36:19.824019 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:19.824022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:19.824035 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:19.824158 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:19.824160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:19.824222 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:19.824298 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:19.824925 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.825460 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.826342 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:19.826556 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:19.826640 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:19.834631 (+ 7991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53985: BlockingRecv error: recv got EOF from 127.0.98.1:53985 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:20.052469 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:20.046469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:51267)
0620 04:36:20.046820 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:20.046836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:20.046996 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:20.047294 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:20.047297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:20.047520 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:20.047616 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.047622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.048484 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.048489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.049105 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.049113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.049624 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.049628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.049719 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.050299 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:20.050312 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:20.052279 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":80,"threads_started":1}
W20260620 04:36:20.052529 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:20.046592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51267 (local address 127.0.98.62:37569)
0620 04:36:20.046787 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:20.046792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:20.046929 (+ 137us) server_negotiation.cc:407] Connection header received
0620 04:36:20.047129 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:20.047132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:20.047183 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:20.047275 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:20.047741 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.048377 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.049233 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.049477 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.049612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.052413 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51267: BlockingRecv error: recv got EOF from 127.0.98.2:51267 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:20.257802 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:20.251550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:59791)
0620 04:36:20.251829 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:20.251842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:20.251938 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:20.252276 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:20.252279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:20.252475 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:20.252572 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.252578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.253344 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.253347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.253948 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.253962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.254599 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.254602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.254710 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.255267 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:20.255285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:20.257632 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
W20260620 04:36:20.257841 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:20.251683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59791 (local address 127.0.98.62:37569)
0620 04:36:20.251830 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:20.251834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:20.251909 (+ 75us) server_negotiation.cc:407] Connection header received
0620 04:36:20.252021 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:20.252023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:20.252068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:20.252140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:20.252689 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.253212 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.254109 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.254426 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.254634 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.257743 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59791: BlockingRecv error: recv got EOF from 127.0.98.3:59791 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:20 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:20.841447 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:20.835532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:57857)
0620 04:36:20.835813 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:20.835827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:20.835908 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:20.836183 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:20.836207 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:20.836418 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:20.836536 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.836542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.837497 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.837500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.838137 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:20.838146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.838663 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.838666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:20.838736 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.839294 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:20.839306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:20.841328 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
W20260620 04:36:20.841670 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:20.835641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57857 (local address 127.0.98.62:37569)
0620 04:36:20.835869 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:20.835872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:20.835885 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:20.835986 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:20.835988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:20.836027 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:20.836102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:20.836690 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.837303 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.838276 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:20.838547 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:20.838651 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:20.841512 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57857: BlockingRecv error: recv got EOF from 127.0.98.1:57857 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:21.059726 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:21.053671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:38919)
0620 04:36:21.053992 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:21.054011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:21.054157 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:21.054436 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:21.054439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:21.054666 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:21.054758 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.054764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.055584 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.055590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.056274 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.056283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.056763 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.056767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.056858 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.057512 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:21.057525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:21.059540 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
W20260620 04:36:21.059789 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:21.053734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38919 (local address 127.0.98.62:37569)
0620 04:36:21.053960 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:21.053965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:21.054097 (+ 132us) server_negotiation.cc:407] Connection header received
0620 04:36:21.054263 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:21.054267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:21.054321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:21.054416 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:21.054880 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.055464 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.056393 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.056641 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.056759 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.059670 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38919: BlockingRecv error: recv got EOF from 127.0.98.2:38919 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:21.264895 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:21.258769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:59231)
0620 04:36:21.259137 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:21.259154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:21.259308 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:21.259609 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:21.259612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:21.259826 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:21.259920 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.259926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.260810 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.260816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.261433 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.261441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.261932 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.261936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.262029 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.262573 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:21.262585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:21.264709 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
W20260620 04:36:21.264945 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:21.258942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59231 (local address 127.0.98.62:37569)
0620 04:36:21.259133 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:21.259137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:21.259247 (+ 110us) server_negotiation.cc:407] Connection header received
0620 04:36:21.259427 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:21.259432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:21.259488 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:21.259585 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:21.260053 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.260693 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.261565 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.261811 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.261928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.264835 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59231: BlockingRecv error: recv got EOF from 127.0.98.3:59231 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:21 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:21.848179 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:21.842358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:51701)
0620 04:36:21.842615 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:21.842628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:21.842715 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:21.842966 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:21.842968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:21.843143 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:21.843233 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.843238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.843997 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.844000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.844639 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:21.844649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.845100 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.845103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:21.845162 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.845778 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:21.845799 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:21.847994 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20260620 04:36:21.848294 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:21.842431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51701 (local address 127.0.98.62:37569)
0620 04:36:21.842664 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:21.842667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:21.842680 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:36:21.842802 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:21.842804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:21.842851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:21.842930 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:21.843344 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.843874 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.844769 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:21.844990 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:21.845079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:21.848123 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51701: BlockingRecv error: recv got EOF from 127.0.98.1:51701 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":55,"threads_started":1}
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:22.066749 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:22.060809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:39143)
0620 04:36:22.061091 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:22.061113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:22.061234 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:22.061674 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:22.061676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:22.061933 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:22.062071 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.062081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.062955 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.062958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.063502 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.063510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.064010 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.064012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.064071 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.064679 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:22.064690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:22.066553 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
W20260620 04:36:22.066787 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:22.060919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39143 (local address 127.0.98.62:37569)
0620 04:36:22.061081 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:22.061087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:22.061192 (+ 105us) server_negotiation.cc:407] Connection header received
0620 04:36:22.061432 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:22.061438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:22.061501 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:22.061628 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:22.062275 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.062855 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.063635 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.063901 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.063999 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.066688 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39143: BlockingRecv error: recv got EOF from 127.0.98.2:39143 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:22.271287 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:22.265714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:45003)
0620 04:36:22.266068 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:22.266081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:22.266221 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:22.266489 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:22.266492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:22.266700 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:22.266789 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.266795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.267543 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.267547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.268102 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.268109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.268608 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.268611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.268692 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.269233 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:22.269245 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:22.271115 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
W20260620 04:36:22.271328 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:22.265837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45003 (local address 127.0.98.62:37569)
0620 04:36:22.266011 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:22.266016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:22.266156 (+ 140us) server_negotiation.cc:407] Connection header received
0620 04:36:22.266318 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:22.266322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:22.266377 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:22.266464 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:22.266906 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.267432 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.268270 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.268492 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.268614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.271229 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45003: BlockingRecv error: recv got EOF from 127.0.98.3:45003 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:22 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:22.855023 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:22.849042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:51655)
0620 04:36:22.849351 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:22.849365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:22.849462 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:22.849752 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:22.849755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:22.849938 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:22.850038 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.850044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.850820 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.850823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.851441 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:22.851450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.851943 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.851946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:22.852030 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.852689 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:22.852702 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:22.854849 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":127,"threads_started":1}
W20260620 04:36:22.855123 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:22.849141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51655 (local address 127.0.98.62:37569)
0620 04:36:22.849363 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:22.849366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:22.849488 (+ 122us) server_negotiation.cc:407] Connection header received
0620 04:36:22.849546 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:22.849548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:22.849593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:22.849673 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:22.850154 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.850694 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.851579 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:22.851809 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:22.851897 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:22.854989 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51655: BlockingRecv error: recv got EOF from 127.0.98.1:51655 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:23.072793 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:23.067534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:49557)
0620 04:36:23.067810 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:23.067823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:23.067919 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:23.068128 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:23.068130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:23.068350 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:23.068443 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.068449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.069239 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.069243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.069817 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.069824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.070276 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.070279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.070342 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.070828 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:23.070845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:23.072682 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
W20260620 04:36:23.073002 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:23.067583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49557 (local address 127.0.98.62:37569)
0620 04:36:23.067726 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:23.067731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:23.067896 (+ 165us) server_negotiation.cc:407] Connection header received
0620 04:36:23.067984 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:23.067986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:23.068029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:23.068094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:23.068564 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.069113 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.069935 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.070158 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.070238 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.072872 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49557: BlockingRecv error: recv got EOF from 127.0.98.2:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:23.278823 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:23.272161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:45483)
0620 04:36:23.272505 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:23.272518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:23.272621 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:23.272863 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:23.272865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:23.273049 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:23.273139 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.273144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.274120 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.274123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.274709 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.274718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.275227 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.275230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.275302 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.276367 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:23.276379 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:23.278706 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
W20260620 04:36:23.279033 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:23.272267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45483 (local address 127.0.98.62:37569)
0620 04:36:23.272430 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:23.272435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:23.272616 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:36:23.272708 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:23.272710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:23.272760 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:23.272831 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:23.273265 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.273969 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.274843 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.275082 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.275174 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.278885 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45483: BlockingRecv error: recv got EOF from 127.0.98.3:45483 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:23 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:23.862326 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:23.856069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:34207)
0620 04:36:23.856395 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:23.856410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:23.856534 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:23.856976 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:23.856979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:23.857252 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:23.857392 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.857401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.858271 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.858277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.858882 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:23.858890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.859403 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.859406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:23.859485 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.860036 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:23.860049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:23.862214 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
W20260620 04:36:23.862623 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:23.856229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34207 (local address 127.0.98.62:37569)
0620 04:36:23.856457 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:23.856463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:23.856478 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:36:23.856663 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:23.856669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:23.856799 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:23.856919 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:23.857556 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.858153 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.859018 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:23.859253 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:23.859349 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:23.862384 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34207: BlockingRecv error: recv got EOF from 127.0.98.1:34207 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:24.081681 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:24.073616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:55285)
0620 04:36:24.073915 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:24.073930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:24.074031 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:24.074301 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:24.074304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:24.074493 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:24.074587 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.074593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.075960 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.075966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.076888 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.076921 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.077573 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.077578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.077720 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.078460 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:24.078478 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:24.081549 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260620 04:36:24.081830 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:24.073775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55285 (local address 127.0.98.62:37569)
0620 04:36:24.073927 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:24.073931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:24.074019 (+ 88us) server_negotiation.cc:407] Connection header received
0620 04:36:24.074109 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:24.074112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:24.074159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:24.074230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:24.074719 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.075733 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.077083 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.077374 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.077527 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.081693 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55285: BlockingRecv error: recv got EOF from 127.0.98.2:55285 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:24.285646 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:24.279649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:52105)
0620 04:36:24.279904 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:24.279917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:24.280015 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:24.280353 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:24.280356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:24.280558 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:24.280664 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.280671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.281496 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.281503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.282201 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.282234 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.282912 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.282915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.282979 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.283492 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:24.283504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:24.285520 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260620 04:36:24.285852 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:24.279724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52105 (local address 127.0.98.62:37569)
0620 04:36:24.279875 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:24.279878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:24.280015 (+ 137us) server_negotiation.cc:407] Connection header received
0620 04:36:24.280097 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:24.280100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:24.280151 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:24.280261 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:24.280826 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.281356 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.282422 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.282778 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.282866 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.285719 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52105: BlockingRecv error: recv got EOF from 127.0.98.3:52105 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:24 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:24.874084 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:24.863109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:60445)
0620 04:36:24.863427 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:24.863442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:24.863573 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:24.863932 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:24.863935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:24.864187 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:24.864333 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.864340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.865252 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.865258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.865812 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:24.865820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.866280 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.866283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:24.866367 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.872129 (+ 5762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:24.872141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:24.873915 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
W20260620 04:36:24.874163 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:24.863227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60445 (local address 127.0.98.62:37569)
0620 04:36:24.863539 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:24.863542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:24.863556 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:24.863716 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:24.863722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:24.863783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:24.863882 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:24.864451 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.865147 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.865931 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:24.866161 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:24.866266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:24.874042 (+ 7776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60445: BlockingRecv error: recv got EOF from 127.0.98.1:60445 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":55,"threads_started":1}
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:25.088712 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:25.082753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:52965)
0620 04:36:25.083051 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:25.083065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:25.083166 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:25.083460 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:25.083463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:25.083655 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:25.083757 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.083762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.084665 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.084668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.085255 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.085263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.085875 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.085879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.085974 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.086615 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:25.086631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:25.088580 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260620 04:36:25.088828 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:25.082830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52965 (local address 127.0.98.62:37569)
0620 04:36:25.082995 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:25.083002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:25.083189 (+ 187us) server_negotiation.cc:407] Connection header received
0620 04:36:25.083238 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:25.083240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:25.083297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:25.083376 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:25.083877 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.084547 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.085389 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.085684 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.085794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.088713 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52965: BlockingRecv error: recv got EOF from 127.0.98.2:52965 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:25.292513 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:25.286504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.3:47183)
0620 04:36:25.286830 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:25.286845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:25.286982 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:25.287255 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:25.287258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:25.287555 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:25.287661 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.287668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.288524 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.288527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.289103 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.289111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.289564 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.289568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.289670 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.290176 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:25.290189 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:25.292363 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
W20260620 04:36:25.292846 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:25.286605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47183 (local address 127.0.98.62:37569)
0620 04:36:25.286735 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:25.286741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:25.286929 (+ 188us) server_negotiation.cc:407] Connection header received
0620 04:36:25.287088 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:25.287091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:25.287146 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:25.287231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:25.287795 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.288406 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.289219 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.289443 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.289548 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.292739 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47183: BlockingRecv error: recv got EOF from 127.0.98.3:47183 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:25 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:25.882045 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:25.875012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.1:52177)
0620 04:36:25.875523 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:25.875554 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:25.875666 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:25.875953 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:25.875961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:25.876426 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:25.876559 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.876566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.877733 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.877735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.878282 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:25.878290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.878801 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.878804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:25.878913 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.879566 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:25.879585 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:25.881856 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
W20260620 04:36:25.882121 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:25.875172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52177 (local address 127.0.98.62:37569)
0620 04:36:25.875442 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:25.875445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:25.875671 (+ 226us) server_negotiation.cc:407] Connection header received
0620 04:36:25.875739 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:25.875741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:25.875789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:25.875883 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:25.876733 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.877557 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.878391 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:25.878647 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:25.878787 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:25.881991 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52177: BlockingRecv error: recv got EOF from 127.0.98.1:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
Jun 20 04:36:26 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:36:26 dist-test-slave-zmf4 krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:36:26.095427 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.089718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37569 (local address 127.0.98.2:43353)
0620 04:36:26.089974 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:26.089987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:26.090112 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:26.090370 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:26.090372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:26.090581 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:26.090665 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.090670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.091428 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.091432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.091980 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.091987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.092506 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.092510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.092592 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.093121 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:26.093134 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:26.095297 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37569: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260620 04:36:26.095558 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:26.089765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43353 (local address 127.0.98.62:37569)
0620 04:36:26.089933 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:26.089938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:26.090070 (+ 132us) server_negotiation.cc:407] Connection header received
0620 04:36:26.090211 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:26.090214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:26.090266 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:26.090350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:26.090780 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.091316 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.092095 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.092398 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.092507 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.095435 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43353: BlockingRecv error: recv got EOF from 127.0.98.2:43353 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:36:26.192135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9119
I20260620 04:36:26.197644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9271
I20260620 04:36:26.203090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9403
I20260620 04:36:26.208262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9049
2026-06-20T04:36:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18128 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T04:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:26Z Disabled control of system clock
I20260620 04:36:26.238134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46449
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:36:26.347039 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:26.347298 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:26.347332 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:26.351212 9599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:26.351262 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:26.351279 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:26.351295 9599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:26.351310 9599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:26.355980 9599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:26.357048 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:26.358330 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:26.366852 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:26.366863 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.366919 9599 server_base.cc:1034] running on GCE node
W20260620 04:36:26.366874 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.367633 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:26.368695 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:26.369911 9599 hybrid_clock.cc:648] HybridClock initialized: now 1781930186369882 us; error 42 us; skew 500 ppm
I20260620 04:36:26.371764 9599 webserver.cc:460] Webserver started at http://127.0.98.62:36437/ using document root <none> and password file <none>
I20260620 04:36:26.372404 9599 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:26.372494 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:26.372718 9599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:26.374990 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "6a775fb7a0874dc18a6ea6147edb2fb5"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "46810b7853eaee9f1c4b331e42ed5d3a"
server_key_iv: "3e3bd28639c7e0aa545d1e717529ef5e"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.375559 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a775fb7a0874dc18a6ea6147edb2fb5"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "46810b7853eaee9f1c4b331e42ed5d3a"
server_key_iv: "3e3bd28639c7e0aa545d1e717529ef5e"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.379225 9599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:36:26.381815 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.383061 9599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:36:26.383208 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "6a775fb7a0874dc18a6ea6147edb2fb5"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "46810b7853eaee9f1c4b331e42ed5d3a"
server_key_iv: "3e3bd28639c7e0aa545d1e717529ef5e"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.383327 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:26.394078 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:26.394688 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:26.394874 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:26.401598 9599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46449
I20260620 04:36:26.401616 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46449 every 8 connection(s)
I20260620 04:36:26.402519 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:26.403923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9599
I20260620 04:36:26.404029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:36:26.404361 392 external_mini_cluster.cc:1428] Setting key 6cab215279c0c4b53661193468c77710
I20260620 04:36:26.406118 9666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:26.411512 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5: Bootstrap starting.
I20260620 04:36:26.413403 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.405855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.0.98.62:46449)
0620 04:36:26.406520 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:26.406536 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:26.406569 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:36:26.407223 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:26.407238 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:26.407593 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:26.407923 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:26.408365 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.409291 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.410152 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.410696 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.410876 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.411733 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:26.411738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:26.411750 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:26.412129 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:26.412181 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:26.412245 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:26.412350 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:26.412507 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 04:36:26.412754 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260620 04:36:26.414054 9666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:26.415020 9666 log.cc:824] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:26.417032 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5: No bootstrap required, opened a new log
I20260620 04:36:26.421075 9666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260620 04:36:26.421291 9666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:26.421339 9666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a775fb7a0874dc18a6ea6147edb2fb5, State: Initialized, Role: FOLLOWER
I20260620 04:36:26.421708 9666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260620 04:36:26.421820 9666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:26.421895 9666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:26.421998 9666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:26.422830 9666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260620 04:36:26.423182 9666 leader_election.cc:302] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a775fb7a0874dc18a6ea6147edb2fb5; no voters:
I20260620 04:36:26.423561 9666 leader_election.cc:288] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:36:26.423636 9671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:26.423900 9671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 6a775fb7a0874dc18a6ea6147edb2fb5, State: Running, Role: LEADER
I20260620 04:36:26.424232 9671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260620 04:36:26.424947 9666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:26.426002 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a775fb7a0874dc18a6ea6147edb2fb5. Latest consensus state: current_term: 1 leader_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } } }
I20260620 04:36:26.426064 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a775fb7a0874dc18a6ea6147edb2fb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } } }
I20260620 04:36:26.426214 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:26.426221 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:26.426664 9680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:26.429965 9680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:26.435743 9680 catalog_manager.cc:1347] Generated new cluster ID: c938e6602e664d9ebe2701b5ccfdbda8
I20260620 04:36:26.435810 9680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:26.456127 9680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:36:26.457430 9680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:26.474723 9680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a775fb7a0874dc18a6ea6147edb2fb5: Generated new TSK 0
I20260620 04:36:26.475601 9680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:36:26.487969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--builtin_ntp_servers=127.0.98.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 04:36:26.609285 9690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:26.609589 9690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:26.609683 9690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:26.614324 9690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:26.614406 9690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:26.614501 9690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:26.619753 9690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:26.620873 9690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:26.622357 9690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:26.632313 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:26.632373 9695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.632412 9690 server_base.cc:1034] running on GCE node
W20260620 04:36:26.632308 9698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.632946 9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:26.633531 9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:26.634713 9690 hybrid_clock.cc:648] HybridClock initialized: now 1781930186634685 us; error 43 us; skew 500 ppm
I20260620 04:36:26.636786 9690 webserver.cc:460] Webserver started at http://127.0.98.1:46677/ using document root <none> and password file <none>
I20260620 04:36:26.637418 9690 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:26.637485 9690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:26.637643 9690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:26.639544 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd32044d521846b38ad4ffa2bc3df80e"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "dcdef675f083c64cd383e87f29346b90"
server_key_iv: "52b9cb07262a1e5bf2f954fa6c27e735"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.640033 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd32044d521846b38ad4ffa2bc3df80e"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "dcdef675f083c64cd383e87f29346b90"
server_key_iv: "52b9cb07262a1e5bf2f954fa6c27e735"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.643636 9690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:26.646284 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.647655 9690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:36:26.647795 9690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "cd32044d521846b38ad4ffa2bc3df80e"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "dcdef675f083c64cd383e87f29346b90"
server_key_iv: "52b9cb07262a1e5bf2f954fa6c27e735"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.647917 9690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:26.664800 9690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:26.665443 9690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:26.665642 9690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:26.666198 9690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:26.667088 9690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:26.667135 9690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.667172 9690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:26.667187 9690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.676017 9690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40969
I20260620 04:36:26.676101 9817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40969 every 8 connection(s)
I20260620 04:36:26.677028 9690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:26.685038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9690
I20260620 04:36:26.685177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:36:26.685510 392 external_mini_cluster.cc:1428] Setting key f6f4dc5fdaa9ec66f9a9c255031e41ba
I20260620 04:36:26.686421 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.678834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51833 (local address 127.0.98.62:46449)
0620 04:36:26.678988 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:26.678993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:26.680275 (+ 1282us) server_negotiation.cc:407] Connection header received
0620 04:36:26.681656 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:26.681659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:26.681719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:26.681840 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:26.682679 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.683217 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.684353 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.684579 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.684686 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.685881 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:26.685885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:26.685887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:26.685898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:26.685916 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:26.685924 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:26.686009 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:26.686206 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 04:36:26.686281 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:36:26.687288 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.679290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.1:51833)
0620 04:36:26.680090 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:26.680161 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:26.681452 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:26.682005 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:26.682011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:26.682090 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:26.682536 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.682557 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.683339 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.683342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.684184 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.684242 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.684720 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.684723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.684892 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.685584 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:26.685586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:26.685632 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:26.685690 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:26.686058 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:26.686065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:26.686069 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:36:26.686205 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 04:36:26.686460 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260620 04:36:26.687559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--builtin_ntp_servers=127.0.98.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:36:26.688668 9818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260620 04:36:26.688885 9818 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:26.689360 9818 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260620 04:36:26.690901 9630 ts_manager.cc:194] Registered new tserver with Master: cd32044d521846b38ad4ffa2bc3df80e (127.0.98.1:40969)
I20260620 04:36:26.692580 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51833
W20260620 04:36:26.801290 9821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:26.801538 9821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:26.801573 9821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:26.805415 9821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:26.805466 9821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:26.805516 9821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:26.810068 9821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:26.810993 9821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:26.812408 9821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:26.821492 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:26.821700 9829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.821722 9821 server_base.cc:1034] running on GCE node
W20260620 04:36:26.821552 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:26.822156 9821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:26.822785 9821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:26.824041 9821 hybrid_clock.cc:648] HybridClock initialized: now 1781930186824018 us; error 32 us; skew 500 ppm
I20260620 04:36:26.826239 9821 webserver.cc:460] Webserver started at http://127.0.98.2:46647/ using document root <none> and password file <none>
I20260620 04:36:26.826887 9821 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:26.827010 9821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:26.827245 9821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:26.829530 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "39ad47fc5387429fbe5eb044981af3b7"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "a2222da00dcd1c63921436a535bbc2e5"
server_key_iv: "003c84d88cbef25e3dddeb267170c6c4"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.830130 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39ad47fc5387429fbe5eb044981af3b7"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "a2222da00dcd1c63921436a535bbc2e5"
server_key_iv: "003c84d88cbef25e3dddeb267170c6c4"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.834347 9821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:36:26.837118 9835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.838335 9821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:26.838482 9821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "39ad47fc5387429fbe5eb044981af3b7"
format_stamp: "Formatted at 2026-06-20 04:36:26 on dist-test-slave-zmf4"
server_key: "a2222da00dcd1c63921436a535bbc2e5"
server_key_iv: "003c84d88cbef25e3dddeb267170c6c4"
server_key_version: "encryptionkey@0"
I20260620 04:36:26.838599 9821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:26.852754 9821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:26.853608 9821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:26.853806 9821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:26.854374 9821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:26.855305 9821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:26.855376 9821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.855440 9821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:26.855470 9821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:26.864530 9821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35761
I20260620 04:36:26.864569 9948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35761 every 8 connection(s)
I20260620 04:36:26.865412 9821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:26.874356 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.867235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48579 (local address 127.0.98.62:46449)
0620 04:36:26.867364 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:26.867368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:26.868342 (+ 974us) server_negotiation.cc:407] Connection header received
0620 04:36:26.869685 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:26.869688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:26.869747 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:26.869851 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:26.870677 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.871202 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.872441 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.872682 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.872795 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.873864 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:26.873867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:26.873870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:26.873889 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:26.873906 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:26.873914 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:26.873991 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:26.874159 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 04:36:26.874217 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:36:26.874961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9821
I20260620 04:36:26.875080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:36:26.875226 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:26.867535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.2:48579)
0620 04:36:26.868149 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:26.868233 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:26.869488 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:26.870006 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:26.870012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:26.870096 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:26.870544 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.870557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.871322 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.871331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.872170 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:26.872178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:26.872834 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:26.872838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:26.873002 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:26.873612 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:26.873613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:26.873651 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:26.873711 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:26.874005 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:26.874015 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:26.874026 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 04:36:26.874145 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 04:36:26.874410 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260620 04:36:26.875404 392 external_mini_cluster.cc:1428] Setting key 8808078a27e73649b83e1c8f1f91e8cf
I20260620 04:36:26.876549 9949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260620 04:36:26.876781 9949 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:26.877292 9949 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260620 04:36:26.877522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--builtin_ntp_servers=127.0.98.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:36:26.878732 9630 ts_manager.cc:194] Registered new tserver with Master: 39ad47fc5387429fbe5eb044981af3b7 (127.0.98.2:35761)
I20260620 04:36:26.879369 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48579
W20260620 04:36:26.985738 9952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:26.986030 9952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:26.986105 9952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:26.990207 9952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:26.990257 9952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:26.990311 9952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:26.994890 9952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:26.995885 9952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:26.997219 9952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:36:27.006457 9952 server_base.cc:1034] running on GCE node
W20260620 04:36:27.006408 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.006405 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.006412 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.007136 9952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:27.007730 9952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:27.008972 9952 hybrid_clock.cc:648] HybridClock initialized: now 1781930187008958 us; error 29 us; skew 500 ppm
I20260620 04:36:27.010713 9952 webserver.cc:460] Webserver started at http://127.0.98.3:35647/ using document root <none> and password file <none>
I20260620 04:36:27.011276 9952 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:27.011336 9952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:27.011543 9952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:27.013651 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "ebd954a310fa499883bcbdc066899f13"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "904c4b263e16a93deaf28006c582e628"
server_key_iv: "a2b46aa4244f1fdf10fafe14ae8f1cca"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.014547 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ebd954a310fa499883bcbdc066899f13"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "904c4b263e16a93deaf28006c582e628"
server_key_iv: "a2b46aa4244f1fdf10fafe14ae8f1cca"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.018383 9952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:36:27.021010 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.022285 9952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:27.022425 9952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "ebd954a310fa499883bcbdc066899f13"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "904c4b263e16a93deaf28006c582e628"
server_key_iv: "a2b46aa4244f1fdf10fafe14ae8f1cca"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.022538 9952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:27.036605 9952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:27.037281 9952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:27.037572 9952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:27.038142 9952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:27.039036 9952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:27.039106 9952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.039163 9952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:27.039214 9952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.047869 9952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35587
I20260620 04:36:27.047891 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35587 every 8 connection(s)
I20260620 04:36:27.048856 9952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:27.053742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 9952
I20260620 04:36:27.053853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:27.054124 392 external_mini_cluster.cc:1428] Setting key ba66610c143c8317c0d8aa2cefa8cc02
I20260620 04:36:27.059051 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.050727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59359 (local address 127.0.98.62:46449)
0620 04:36:27.050906 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.050911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.052081 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 04:36:27.053568 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.053575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.053648 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.053760 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.054804 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:27.055642 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:27.057058 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:27.057266 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:27.057356 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:27.058581 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:27.058584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:27.058586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:27.058605 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:27.058622 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:27.058630 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:27.058692 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:27.058859 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 04:36:27.058925 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:36:27.059824 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.051190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.3:59359)
0620 04:36:27.051927 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:27.051977 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:27.053326 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:27.053991 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:27.053997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:27.054076 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:27.054666 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:27.054681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:27.055782 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:27.055785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:27.056928 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:27.056937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:27.057443 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:27.057446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:27.057599 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:27.058348 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:27.058350 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:27.058388 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:27.058446 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:27.058707 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:27.058713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:27.058717 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:36:27.058808 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 04:36:27.059011 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":118,"threads_started":1}
I20260620 04:36:27.060985 10080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260620 04:36:27.061216 10080 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:27.061714 10080 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260620 04:36:27.062736 9630 ts_manager.cc:194] Registered new tserver with Master: ebd954a310fa499883bcbdc066899f13 (127.0.98.3:35587)
I20260620 04:36:27.063424 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59359
I20260620 04:36:27.068325 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:36:27.070966 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:27.070181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.98.62:46449)
0620 04:36:27.070299 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.070303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.070499 (+ 196us) server_negotiation.cc:407] Connection header received
0620 04:36:27.070565 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.070567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.070604 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.070673 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.070878 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37376: BlockingRecv error: recv got EOF from 127.0.0.1:37376 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:36:27.071187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9690
I20260620 04:36:27.076607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9821
I20260620 04:36:27.081951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9952
I20260620 04:36:27.087127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 9599
2026-06-20T04:36:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (877 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T04:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:27Z Disabled control of system clock
I20260620 04:36:27.116084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41207
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:36:27.224924 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:27.225199 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:27.225239 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:27.229694 10091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:27.229760 10091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:27.229777 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:27.229795 10091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:27.229812 10091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:27.235038 10091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:27.236099 10091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:27.237527 10091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:27.246281 10097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.246438 10091 server_base.cc:1034] running on GCE node
W20260620 04:36:27.246328 10096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.246295 10099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.247076 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:27.248143 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:27.249383 10091 hybrid_clock.cc:648] HybridClock initialized: now 1781930187249356 us; error 40 us; skew 500 ppm
I20260620 04:36:27.251497 10091 webserver.cc:460] Webserver started at http://127.0.98.62:36943/ using document root <none> and password file <none>
I20260620 04:36:27.252080 10091 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:27.252185 10091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:27.252441 10091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:27.254449 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "6e17c5728e004cd59951ece3cd348ca9"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "7426baaa40c6fadb30beee67972d8271"
server_key_iv: "71426b96cbc097ddc803c282dd00ca8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.255018 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e17c5728e004cd59951ece3cd348ca9"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "7426baaa40c6fadb30beee67972d8271"
server_key_iv: "71426b96cbc097ddc803c282dd00ca8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.258806 10091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:36:27.261373 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.262672 10091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:27.262825 10091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "6e17c5728e004cd59951ece3cd348ca9"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "7426baaa40c6fadb30beee67972d8271"
server_key_iv: "71426b96cbc097ddc803c282dd00ca8f"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.262945 10091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:27.267536 10091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:27.268117 10091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:27.268332 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:27.275476 10091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41207
I20260620 04:36:27.275542 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41207 every 8 connection(s)
I20260620 04:36:27.276512 10091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:27.279774 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:27.281756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10091
I20260620 04:36:27.281904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:36:27.282182 392 external_mini_cluster.cc:1428] Setting key 5e0c90806aecd0f11a94c44dbd07a85b
I20260620 04:36:27.285617 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9: Bootstrap starting.
I20260620 04:36:27.287832 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.283553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.0.98.62:41207)
0620 04:36:27.284292 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.284309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.284346 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:36:27.285030 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.285048 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.285400 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.285751 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.286644 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:27.286648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:27.286660 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:27.286670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:27.286723 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:27.286738 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:27.286886 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:27.286956 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 04:36:27.287190 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260620 04:36:27.287945 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:27.288754 10158 log.cc:824] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:27.290753 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9: No bootstrap required, opened a new log
I20260620 04:36:27.294518 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } }
I20260620 04:36:27.294730 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:27.294762 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e17c5728e004cd59951ece3cd348ca9, State: Initialized, Role: FOLLOWER
I20260620 04:36:27.295200 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } }
I20260620 04:36:27.295320 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:27.295357 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:27.295446 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:27.296424 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } }
I20260620 04:36:27.296758 10158 leader_election.cc:302] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e17c5728e004cd59951ece3cd348ca9; no voters:
I20260620 04:36:27.297078 10158 leader_election.cc:288] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:36:27.297267 10163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:27.297479 10163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 6e17c5728e004cd59951ece3cd348ca9, State: Running, Role: LEADER
I20260620 04:36:27.297813 10163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } }
I20260620 04:36:27.298487 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:27.299618 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e17c5728e004cd59951ece3cd348ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } } }
I20260620 04:36:27.299746 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:27.300002 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e17c5728e004cd59951ece3cd348ca9. Latest consensus state: current_term: 1 leader_uuid: "6e17c5728e004cd59951ece3cd348ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e17c5728e004cd59951ece3cd348ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41207 } } }
I20260620 04:36:27.300092 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:27.300231 10172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:27.303329 10172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:27.309554 10172 catalog_manager.cc:1347] Generated new cluster ID: 0d1bfdab0bb54f2885b91405deff996e
I20260620 04:36:27.309706 10172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:27.363114 10172 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:36:27.363471 10172 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:27.368093 10172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e17c5728e004cd59951ece3cd348ca9: Generated new TSK 0
I20260620 04:36:27.368762 10172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:36:27.379863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--builtin_ntp_servers=127.0.98.20:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:36:27.510775 10182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:27.511176 10182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:27.511257 10182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:27.516384 10182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:27.516454 10182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:27.516522 10182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:27.522051 10182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:27.523022 10182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:27.524307 10182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:36:27.533653 10182 server_base.cc:1034] running on GCE node
W20260620 04:36:27.533627 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.533605 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.533965 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.534410 10182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:27.535032 10182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:27.536264 10182 hybrid_clock.cc:648] HybridClock initialized: now 1781930187536224 us; error 42 us; skew 500 ppm
I20260620 04:36:27.538144 10182 webserver.cc:460] Webserver started at http://127.0.98.1:40735/ using document root <none> and password file <none>
I20260620 04:36:27.538702 10182 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:27.538763 10182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:27.538966 10182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:27.541191 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "452d62ff27254baaadc28efdb7928b06"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "6489ca237ca0b4cec20fb18e7dbd2b01"
server_key_iv: "1eb2bd8ea5b4b52732e87fe1b4147242"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.541797 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "452d62ff27254baaadc28efdb7928b06"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "6489ca237ca0b4cec20fb18e7dbd2b01"
server_key_iv: "1eb2bd8ea5b4b52732e87fe1b4147242"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.545549 10182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:27.547994 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.549269 10182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:27.549499 10182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "452d62ff27254baaadc28efdb7928b06"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "6489ca237ca0b4cec20fb18e7dbd2b01"
server_key_iv: "1eb2bd8ea5b4b52732e87fe1b4147242"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.549633 10182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:27.554100 10182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:27.554664 10182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:27.554852 10182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:27.555370 10182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:27.556303 10182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:27.556370 10182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.556430 10182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:27.556480 10182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.566187 10182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39225
I20260620 04:36:27.566270 10309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39225 every 8 connection(s)
I20260620 04:36:27.567229 10182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:27.574165 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.569170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35613 (local address 127.0.98.62:41207)
0620 04:36:27.569331 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.569337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.570570 (+ 1233us) server_negotiation.cc:407] Connection header received
0620 04:36:27.572012 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.572016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.572055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.572333 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.573638 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:27.573642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:27.573644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:27.573656 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:27.573677 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:27.573686 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:27.573791 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:27.573995 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 04:36:27.574042 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:36:27.575038 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.569682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41207 (local address 127.0.98.1:35613)
0620 04:36:27.570411 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:27.570450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:27.571763 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:27.572525 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:27.572531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:27.572618 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:27.573208 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:27.573210 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:27.573251 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:27.573471 (+ 220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:27.573812 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:27.573820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:27.573824 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:36:27.573976 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 04:36:27.574220 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260620 04:36:27.576663 10310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41207
I20260620 04:36:27.577137 10310 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:27.577270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10182
I20260620 04:36:27.577394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:36:27.577633 10310 heartbeater.cc:511] Master 127.0.98.62:41207 requested a full tablet report, sending...
I20260620 04:36:27.577702 392 external_mini_cluster.cc:1428] Setting key 4ea3e009568a9ee4e8259ba45797012b
I20260620 04:36:27.579543 10121 ts_manager.cc:194] Registered new tserver with Master: 452d62ff27254baaadc28efdb7928b06 (127.0.98.1:39225)
I20260620 04:36:27.579892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--builtin_ntp_servers=127.0.98.20:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:36:27.688907 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:27.689157 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:27.689196 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:27.693099 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:27.693141 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:27.693197 10313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:27.698072 10313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:27.699183 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:27.700703 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:27.710001 10318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.709944 10321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.710161 10313 server_base.cc:1034] running on GCE node
W20260620 04:36:27.709961 10319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.710579 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:27.711182 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:27.712571 10313 hybrid_clock.cc:648] HybridClock initialized: now 1781930187712543 us; error 70 us; skew 500 ppm
I20260620 04:36:27.714534 10313 webserver.cc:460] Webserver started at http://127.0.98.2:45397/ using document root <none> and password file <none>
I20260620 04:36:27.715106 10313 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:27.715206 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:27.715435 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:27.717483 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "97e01abca1484aafa5ab33272aa034b8"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "3e21f3c00152ea890739722a0b6d4afc"
server_key_iv: "83a5d91c6764e4879308c375051bb3a6"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.718039 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97e01abca1484aafa5ab33272aa034b8"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "3e21f3c00152ea890739722a0b6d4afc"
server_key_iv: "83a5d91c6764e4879308c375051bb3a6"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.721589 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:27.723915 10327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.725072 10313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:36:27.725203 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "97e01abca1484aafa5ab33272aa034b8"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "3e21f3c00152ea890739722a0b6d4afc"
server_key_iv: "83a5d91c6764e4879308c375051bb3a6"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.725309 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:27.729700 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:27.730293 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:27.730420 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:27.730872 10313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:27.732376 10313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:27.732435 10313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.732475 10313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:27.732491 10313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.741374 10313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40355
I20260620 04:36:27.741393 10440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40355 every 8 connection(s)
I20260620 04:36:27.742386 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:27.746384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10313
I20260620 04:36:27.746563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:36:27.746912 392 external_mini_cluster.cc:1428] Setting key 140bd9ea2b78c0a32d135800214760d6
I20260620 04:36:27.748900 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.744370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59185 (local address 127.0.98.62:41207)
0620 04:36:27.744529 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.744535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.745639 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 04:36:27.746933 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.746937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.746984 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.747091 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.748301 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:27.748304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:27.748306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:27.748318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:27.748338 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:27.748346 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:27.748452 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:27.748719 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 04:36:27.748767 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:36:27.749325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--builtin_ntp_servers=127.0.98.20:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 04:36:27.749734 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.744858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41207 (local address 127.0.98.2:59185)
0620 04:36:27.745485 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:27.745523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:27.746731 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:27.747233 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:27.747239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:27.747350 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:27.747952 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:27.747954 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:27.748005 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:27.748084 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:27.748447 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:27.748456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:27.748461 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:36:27.748667 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 04:36:27.748880 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260620 04:36:27.751055 10441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41207
I20260620 04:36:27.751292 10441 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:27.751691 10441 heartbeater.cc:511] Master 127.0.98.62:41207 requested a full tablet report, sending...
I20260620 04:36:27.752840 10121 ts_manager.cc:194] Registered new tserver with Master: 97e01abca1484aafa5ab33272aa034b8 (127.0.98.2:40355)
W20260620 04:36:27.858047 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:27.858278 10444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:27.858315 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:27.862206 10444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:27.862252 10444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:27.862308 10444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:27.866937 10444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:27.867945 10444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:27.869267 10444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:27.878298 10449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.878298 10450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:27.878309 10452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:27.878607 10444 server_base.cc:1034] running on GCE node
I20260620 04:36:27.879063 10444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:27.879652 10444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:27.880836 10444 hybrid_clock.cc:648] HybridClock initialized: now 1781930187880806 us; error 50 us; skew 500 ppm
I20260620 04:36:27.882650 10444 webserver.cc:460] Webserver started at http://127.0.98.3:43195/ using document root <none> and password file <none>
I20260620 04:36:27.883200 10444 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:27.883291 10444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:27.883512 10444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:27.885547 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "619c2e17bbae45a287c00c52ff64d113"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "a3f49a4ae75819a73cd2a758f751f229"
server_key_iv: "2980af9653446a13595f85ce8d84aa36"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.886096 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "619c2e17bbae45a287c00c52ff64d113"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "a3f49a4ae75819a73cd2a758f751f229"
server_key_iv: "2980af9653446a13595f85ce8d84aa36"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.889647 10444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:36:27.891933 10458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.893311 10444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:36:27.893482 10444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "619c2e17bbae45a287c00c52ff64d113"
format_stamp: "Formatted at 2026-06-20 04:36:27 on dist-test-slave-zmf4"
server_key: "a3f49a4ae75819a73cd2a758f751f229"
server_key_iv: "2980af9653446a13595f85ce8d84aa36"
server_key_version: "encryptionkey@0"
I20260620 04:36:27.893604 10444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:27.898425 10444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:27.899071 10444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:27.899293 10444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:27.899891 10444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:27.900889 10444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:27.900965 10444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.901038 10444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:27.901096 10444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:27.910595 10444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43555
I20260620 04:36:27.910625 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43555 every 8 connection(s)
I20260620 04:36:27.911491 10444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:27.916230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10444
I20260620 04:36:27.916342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:27.916640 392 external_mini_cluster.cc:1428] Setting key 89deb060cd72338d16f88d72dd7bd803
I20260620 04:36:27.918036 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.913437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34117 (local address 127.0.98.62:41207)
0620 04:36:27.913612 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.913617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.914830 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 04:36:27.916050 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.916054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.916102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.916241 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.917541 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:27.917546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:27.917549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:36:27.917563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:27.917587 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:36:27.917599 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:27.917691 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:27.917862 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 04:36:27.917907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:36:27.919061 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:27.913985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41207 (local address 127.0.98.3:34117)
0620 04:36:27.914694 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:27.914730 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:27.915839 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:27.916459 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:27.916466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:27.916546 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:27.917191 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:36:27.917194 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:27.917245 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:36:27.917349 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:27.917700 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:27.917709 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:27.917715 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:36:27.917895 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 04:36:27.918162 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":153,"threads_started":1}
I20260620 04:36:27.920081 10572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41207
I20260620 04:36:27.920389 10572 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:27.920845 10572 heartbeater.cc:511] Master 127.0.98.62:41207 requested a full tablet report, sending...
I20260620 04:36:27.921818 10121 ts_manager.cc:194] Registered new tserver with Master: 619c2e17bbae45a287c00c52ff64d113 (127.0.98.3:43555)
I20260620 04:36:27.930675 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:36:27.933733 10161 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:36:27.932764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.0.98.62:41207)
0620 04:36:27.932892 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:27.932895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:27.933067 (+ 172us) server_negotiation.cc:407] Connection header received
0620 04:36:27.933144 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:27.933146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:27.933177 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:27.933254 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:27.933581 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57070: BlockingRecv error: recv got EOF from 127.0.0.1:57070 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:36:27.933851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10182
I20260620 04:36:27.939092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10313
I20260620 04:36:27.944090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10444
I20260620 04:36:27.948995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10091
2026-06-20T04:36:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (861 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:36:28 dist-test-slave-zmf4 krb5kdc[10582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:36:28 dist-test-slave-zmf4 krb5kdc[10582](info): set up 2 sockets
Jun 20 04:36:28 dist-test-slave-zmf4 krb5kdc[10582](info): commencing operation
krb5kdc: starting...
W20260620 04:36:29.042481 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:36:29.171726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:29.284128 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:29.284490 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:29.284533 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:29.288616 10595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:36:29.288672 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:29.288692 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:29.288709 10595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:36:29.288755 10595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:36:29.293999 10595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:29.295032 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:29.296025 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:29.304821 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.304854 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.304819 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:29.305033 10595 server_base.cc:1034] running on GCE node
I20260620 04:36:29.306109 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:29.307299 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:29.308521 10595 hybrid_clock.cc:648] HybridClock initialized: now 1781930189308505 us; error 33 us; skew 500 ppm
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:29.311260 10595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:36:29.312269 10595 webserver.cc:460] Webserver started at http://127.0.98.62:43163/ using document root <none> and password file <none>
I20260620 04:36:29.312954 10595 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:29.313062 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:29.313304 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:29.315316 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "1b23376d96584ef085303abbba27a8cc"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "bf748bdb7f0bb9dc927ad8cdec7f7973"
server_key_iv: "47a4232550450afd36c0e7934d49e27f"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.315881 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b23376d96584ef085303abbba27a8cc"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "bf748bdb7f0bb9dc927ad8cdec7f7973"
server_key_iv: "47a4232550450afd36c0e7934d49e27f"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.319741 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 04:36:29.322220 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.324005 10595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:29.324152 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "1b23376d96584ef085303abbba27a8cc"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "bf748bdb7f0bb9dc927ad8cdec7f7973"
server_key_iv: "47a4232550450afd36c0e7934d49e27f"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.324384 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:29.341259 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:29.344535 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:29.344729 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:29.351364 10595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46069
I20260620 04:36:29.351637 10662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46069 every 8 connection(s)
I20260620 04:36:29.352430 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:36:29.355458 10663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:29.358127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10595
I20260620 04:36:29.358268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:36:29.358534 392 external_mini_cluster.cc:1428] Setting key 955ea1f1552193f6b850f2e7c6555359
I20260620 04:36:29.362257 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: Bootstrap starting.
I20260620 04:36:29.365044 10663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:29.365839 10663 log.cc:824] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:29.367928 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: No bootstrap required, opened a new log
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:29.371402 10663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260620 04:36:29.371613 10663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:29.371649 10663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b23376d96584ef085303abbba27a8cc, State: Initialized, Role: FOLLOWER
I20260620 04:36:29.372294 10663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260620 04:36:29.372715 10663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:36:29.372824 10663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:36:29.372928 10663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:29.374408 10663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260620 04:36:29.374816 10663 leader_election.cc:302] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b23376d96584ef085303abbba27a8cc; no voters:
I20260620 04:36:29.375201 10663 leader_election.cc:288] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:36:29.375276 10668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:29.375492 10668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [term 1 LEADER]: Becoming Leader. State: Replica: 1b23376d96584ef085303abbba27a8cc, State: Running, Role: LEADER
I20260620 04:36:29.375712 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:29.359987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.98.62:46069)
0620 04:36:29.360452 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:29.360472 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:29.360516 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:36:29.361423 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:29.361447 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:29.361871 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:29.362251 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:29.363341 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.365010 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.366068 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.366531 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.366697 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:29.369501 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:29.369506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:29.369519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:29.369545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:29.371873 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.372842 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.372851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.372859 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.373022 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.373530 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.373532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.373534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.373922 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:29.374081 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:29.374453 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 04:36:29.374785 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
I20260620 04:36:29.375805 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260620 04:36:29.376565 10663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:36:29.377673 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b23376d96584ef085303abbba27a8cc. Latest consensus state: current_term: 1 leader_uuid: "1b23376d96584ef085303abbba27a8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } } }
I20260620 04:36:29.377816 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:29.378139 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b23376d96584ef085303abbba27a8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b23376d96584ef085303abbba27a8cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } } }
I20260620 04:36:29.378217 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc [sys.catalog]: This master's current role is: LEADER
I20260620 04:36:29.378484 10677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:36:29.381245 10677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:36:29.386559 10677 catalog_manager.cc:1347] Generated new cluster ID: 3fab746f5cef411f8850ba0bd9322f4a
I20260620 04:36:29.386629 10677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:36:29.411433 10677 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:36:29.412827 10677 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:36:29.417553 10677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: Generated new TSK 0
I20260620 04:36:29.418632 10677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:36:29.447530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:29.558830 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:29.559077 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:29.559118 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:29.563169 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:29.563227 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:29.563283 10689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:36:29.568508 10689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:29.569484 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:29.570465 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:29.580073 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.580080 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.580181 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:29.580430 10689 server_base.cc:1034] running on GCE node
I20260620 04:36:29.580802 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:29.581439 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:29.582649 10689 hybrid_clock.cc:648] HybridClock initialized: now 1781930189582615 us; error 46 us; skew 500 ppm
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:29.585469 10689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:36:29.586337 10689 webserver.cc:460] Webserver started at http://127.0.98.1:35325/ using document root <none> and password file <none>
I20260620 04:36:29.586827 10689 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:29.586891 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:29.587049 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:29.589030 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "a5fef56473f543bf8fe1c27f0833732c"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "dc72d7cea2d548004d2e617a738b35aa"
server_key_iv: "bef3360ea7e3e73a93d58d4d0aa11311"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.589533 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a5fef56473f543bf8fe1c27f0833732c"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "dc72d7cea2d548004d2e617a738b35aa"
server_key_iv: "bef3360ea7e3e73a93d58d4d0aa11311"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.593072 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:36:29.595407 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.596951 10689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:29.597051 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "a5fef56473f543bf8fe1c27f0833732c"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "dc72d7cea2d548004d2e617a738b35aa"
server_key_iv: "bef3360ea7e3e73a93d58d4d0aa11311"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.597193 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:29.620998 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:29.624336 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:29.624543 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:29.625219 10689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:29.626046 10689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:29.626094 10689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.626168 10689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:29.626199 10689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.635814 10689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37515
I20260620 04:36:29.635905 10817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37515 every 8 connection(s)
I20260620 04:36:29.636765 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:36:29.643843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10689
I20260620 04:36:29.643962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:36:29.644315 392 external_mini_cluster.cc:1428] Setting key f658fde488ff622a67044b5059a11f80
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:29.650965 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:29.638713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43081 (local address 127.0.98.62:46069)
0620 04:36:29.638850 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:29.638854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:29.639654 (+ 800us) server_negotiation.cc:407] Connection header received
0620 04:36:29.640957 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:29.640960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:29.641016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:29.641097 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:29.642208 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.642754 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.644126 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.644499 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.644590 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:29.647181 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:29.647189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:29.647191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:29.647213 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:29.648793 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.649343 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.649348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.649351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.649438 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.649800 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.649805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.649807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.650005 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:29.650122 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:29.650631 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 04:36:29.650763 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:36:29.652045 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:29.638997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46069 (local address 127.0.98.1:43081)
0620 04:36:29.639499 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:29.639542 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:29.640754 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:29.641261 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:29.641268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:29.641609 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:29.642062 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:29.642075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.642886 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.642894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:29.643924 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:29.643938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.644655 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.644659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:29.644824 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:29.645474 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:29.645490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:29.647008 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:29.648925 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:29.648931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:29.648948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:29.649228 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:29.649545 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:29.649549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:29.649551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:29.649697 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:29.650120 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:29.650127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:29.650378 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 04:36:29.650589 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 04:36:29.650925 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:36:29.653643 10818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46069
I20260620 04:36:29.653913 10818 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:29.654456 10818 heartbeater.cc:511] Master 127.0.98.62:46069 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:36:29.656874 10626 ts_manager.cc:194] Registered new tserver with Master: a5fef56473f543bf8fe1c27f0833732c (127.0.98.1:37515)
I20260620 04:36:29.658026 10626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:36:29.675194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:29.787439 10823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:29.787699 10823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:29.787743 10823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:29.791952 10823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:29.792014 10823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:29.792075 10823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:36:29.797667 10823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:29.798712 10823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:29.799705 10823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:36:29.809036 10831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.809058 10829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:29.809059 10828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:29.809284 10823 server_base.cc:1034] running on GCE node
I20260620 04:36:29.809687 10823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:29.810320 10823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:29.811492 10823 hybrid_clock.cc:648] HybridClock initialized: now 1781930189811476 us; error 29 us; skew 500 ppm
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:29.814422 10823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:36:29.815330 10823 webserver.cc:460] Webserver started at http://127.0.98.2:39127/ using document root <none> and password file <none>
I20260620 04:36:29.815882 10823 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:29.815984 10823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:29.816272 10823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:29.818697 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "e3ad815e7030d87ccf33fb39cc5a9f52"
server_key_iv: "f04c6f02342ef4af6f6fc76ad4be3d42"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.819319 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "e3ad815e7030d87ccf33fb39cc5a9f52"
server_key_iv: "f04c6f02342ef4af6f6fc76ad4be3d42"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.823339 10823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 04:36:29.826135 10838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.827484 10823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:29.827636 10823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199"
format_stamp: "Formatted at 2026-06-20 04:36:29 on dist-test-slave-zmf4"
server_key: "e3ad815e7030d87ccf33fb39cc5a9f52"
server_key_iv: "f04c6f02342ef4af6f6fc76ad4be3d42"
server_key_version: "encryptionkey@0"
I20260620 04:36:29.827751 10823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:29.838229 10823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:29.841295 10823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:29.841499 10823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:29.842067 10823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:29.842860 10823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:29.842932 10823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.842999 10823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:29.843051 10823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:29.852274 10823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34207
I20260620 04:36:29.852336 10951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34207 every 8 connection(s)
I20260620 04:36:29.853174 10823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:36:29.862350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10823
I20260620 04:36:29.862447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
Jun 20 04:36:29 dist-test-slave-zmf4 krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:29.862771 392 external_mini_cluster.cc:1428] Setting key c987ab745a1af256e519d113e670b578
I20260620 04:36:29.866648 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:29.854996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50623 (local address 127.0.98.62:46069)
0620 04:36:29.855129 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:29.855133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:29.856174 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 04:36:29.857283 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:29.857287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:29.857419 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:29.857504 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:29.858653 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.859182 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.860324 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.860545 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.860636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:29.863162 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:29.863166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:29.863168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:29.863191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:29.864652 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.865174 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.865179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.865181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.865260 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:29.865589 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:29.865594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:29.865596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:29.865772 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:29.865878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:29.866356 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 04:36:29.866473 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:36:29.867565 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:29.855366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46069 (local address 127.0.98.2:50623)
0620 04:36:29.856033 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:29.856072 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:29.856896 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:29.857700 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:29.857706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:29.858054 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:29.858493 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:29.858505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.859299 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.859302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:29.860166 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:29.860174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:29.860687 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:29.860690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:29.860832 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:29.861418 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:29.861433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:29.862984 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:29.864783 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:29.864789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:29.864803 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:29.865068 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:29.865349 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:29.865353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:29.865355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:29.865494 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:29.865875 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:29.865881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:29.866127 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 04:36:29.866315 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 04:36:29.866619 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":117,"threads_started":1}
I20260620 04:36:29.868706 10952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46069
I20260620 04:36:29.868949 10952 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:29.869418 10952 heartbeater.cc:511] Master 127.0.98.62:46069 requested a full tablet report, sending...
I20260620 04:36:29.870635 10626 ts_manager.cc:194] Registered new tserver with Master: a1d6ff9ac9a64f118aebad4a2ab1a199 (127.0.98.2:34207)
I20260620 04:36:29.871484 10626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50623
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:36:29.892036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:36:30.004065 10957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:36:30.004366 10957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:36:30.004412 10957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:36:30.008524 10957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:36:30.008584 10957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:36:30.008646 10957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:36:30.014163 10957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:36:30.015156 10957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:36:30.016103 10957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:36:30.025781 10957 server_base.cc:1034] running on GCE node
W20260620 04:36:30.025745 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:30.025764 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:36:30.025738 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:36:30.026389 10957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:36:30.027053 10957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:36:30.028311 10957 hybrid_clock.cc:648] HybridClock initialized: now 1781930190028283 us; error 37 us; skew 500 ppm
Jun 20 04:36:30 dist-test-slave-zmf4 krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:36:30.034168 10957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:36:30.035173 10957 webserver.cc:460] Webserver started at http://127.0.98.3:40765/ using document root <none> and password file <none>
I20260620 04:36:30.035797 10957 fs_manager.cc:359] Metadata directory not provided
I20260620 04:36:30.035864 10957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:36:30.036091 10957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:36:30.038532 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca97aa2f6c48413a8dee2cabcf075e72"
format_stamp: "Formatted at 2026-06-20 04:36:30 on dist-test-slave-zmf4"
server_key: "ba1b0109fc7fb6f648cdca7e8ed912f6"
server_key_iv: "be81f8c9d984c6adb14d0f14ddd9fdfc"
server_key_version: "encryptionkey@0"
I20260620 04:36:30.039419 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca97aa2f6c48413a8dee2cabcf075e72"
format_stamp: "Formatted at 2026-06-20 04:36:30 on dist-test-slave-zmf4"
server_key: "ba1b0109fc7fb6f648cdca7e8ed912f6"
server_key_iv: "be81f8c9d984c6adb14d0f14ddd9fdfc"
server_key_version: "encryptionkey@0"
I20260620 04:36:30.043553 10957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:36:30.046626 10972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:30.048039 10957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:36:30.048137 10957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "ca97aa2f6c48413a8dee2cabcf075e72"
format_stamp: "Formatted at 2026-06-20 04:36:30 on dist-test-slave-zmf4"
server_key: "ba1b0109fc7fb6f648cdca7e8ed912f6"
server_key_iv: "be81f8c9d984c6adb14d0f14ddd9fdfc"
server_key_version: "encryptionkey@0"
I20260620 04:36:30.048288 10957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:36:30.073426 10957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:36:30.077000 10957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:36:30.077165 10957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:36:30.077965 10957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:36:30.079234 10957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:36:30.079289 10957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:30.079331 10957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:36:30.079347 10957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:36:30.089826 10957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39827
I20260620 04:36:30.089843 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39827 every 8 connection(s)
I20260620 04:36:30.090823 10957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:36:30.099264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 10957
I20260620 04:36:30.099470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:36:30.099874 392 external_mini_cluster.cc:1428] Setting key 90312b23d6559cdc62e7e054a4f338dc
Jun 20 04:36:30 dist-test-slave-zmf4 krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:36:30.108312 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.093184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39763 (local address 127.0.98.62:46069)
0620 04:36:30.093369 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.093373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.094643 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 04:36:30.095604 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.095608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.095654 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.095734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:30.097151 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.097981 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.100651 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.101133 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.101297 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.104483 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:30.104487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:30.104489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:30.104515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:30.106004 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:30.106580 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:30.106582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:30.106584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:30.106677 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:30.107091 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:30.107094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:30.107095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:30.107319 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:30.107409 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.107975 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.108098 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 04:36:30.109083 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.093538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46069 (local address 127.0.98.3:39763)
0620 04:36:30.094489 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.094531 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.095373 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.095930 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.095949 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.096438 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:36:30.096978 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.096993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.098952 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.098957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.100387 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.100408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.101359 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.101366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.101583 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.102422 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:36:30.102439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:36:30.104276 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:36:30.106146 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:30.106151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:30.106165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:30.106444 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:30.106783 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:36:30.106786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:36:30.106787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:36:30.106942 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:36:30.107425 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:36:30.107430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:36:30.107708 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.107865 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.108140 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":112,"threads_started":1}
I20260620 04:36:30.110302 11086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46069
I20260620 04:36:30.110522 11086 heartbeater.cc:464] Registering TS with master...
I20260620 04:36:30.110973 11086 heartbeater.cc:511] Master 127.0.98.62:46069 requested a full tablet report, sending...
I20260620 04:36:30.112299 10626 ts_manager.cc:194] Registered new tserver with Master: ca97aa2f6c48413a8dee2cabcf075e72 (127.0.98.3:39827)
I20260620 04:36:30.113039 10626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39763
I20260620 04:36:30.114467 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:36:30.125717 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.116805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.98.62:46069)
0620 04:36:30.116923 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.116926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.116992 (+ 66us) server_negotiation.cc:407] Connection header received
0620 04:36:30.117110 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.117112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.117150 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.117223 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:36:30.118416 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.119185 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.120163 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.120598 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.120693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.122434 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:36:30.122438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:36:30.122440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:36:30.122462 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:36:30.123914 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:30.124346 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:30.124348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:30.124349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:30.124419 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:36:30.124741 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:36:30.124744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:36:30.124745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:36:30.124961 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:36:30.125076 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.125382 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.125535 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:36:30.129315 10626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:36:30.133613 10626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:36:30.148957 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.142936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.0.98.3:39827)
0620 04:36:30.144095 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.144107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.144128 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:36:30.144215 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.144222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.144387 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.144518 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:30.144905 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.145717 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.147364 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.148228 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.148355 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.148506 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.148689 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.148801 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":114,"threads_started":1}
I20260620 04:36:30.148963 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.143225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34207 (local address 127.0.0.1:46230)
0620 04:36:30.143696 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.143707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.143803 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.144538 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.144544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.144570 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:30.144698 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.144716 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.146204 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.146207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.147382 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.147390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.148572 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.148575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.148679 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.148705 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.148775 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.148836 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":51,"thread_start_us":103,"threads_started":1}
I20260620 04:36:30.149283 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.143448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.2:34207)
0620 04:36:30.143832 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.143844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.143867 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:36:30.143943 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.143948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.144153 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.144324 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:30.144841 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.146066 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.147502 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.148289 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.148374 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.148502 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.149040 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.149146 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":96,"threads_started":1}
I20260620 04:36:30.148944 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.142796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39827 (local address 127.0.0.1:38060)
0620 04:36:30.143144 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.143196 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.143333 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.144656 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.144659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.144673 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:30.144758 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.144766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.145853 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.145856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.147186 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.147196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.148355 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.148361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.148476 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.148523 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.148648 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.148773 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260620 04:36:30.149549 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.143457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37515 (local address 127.0.0.1:35110)
0620 04:36:30.144288 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.144302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.144397 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.144916 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.144918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.144936 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:30.145014 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.145020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.146311 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.146315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.147714 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.147722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.149158 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.149161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.149247 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.149268 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.149344 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.149420 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260620 04:36:30.150410 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.143675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.98.1:37515)
0620 04:36:30.143960 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.143973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.144489 (+ 516us) server_negotiation.cc:407] Connection header received
0620 04:36:30.144567 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.144574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.144754 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.144932 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:30.145101 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.146185 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.147831 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.148566 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.148669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.148793 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.150144 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.150248 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260620 04:36:30.152591 10886 tablet_service.cc:1467] Processing CreateTablet for tablet 2344a170834645609b6a0774aa79a709 (DEFAULT_TABLE table=test-table [id=80a39bc6485940d1affd50634eee5efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:30.153407 11020 tablet_service.cc:1467] Processing CreateTablet for tablet 2344a170834645609b6a0774aa79a709 (DEFAULT_TABLE table=test-table [id=80a39bc6485940d1affd50634eee5efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:30.153205 10752 tablet_service.cc:1467] Processing CreateTablet for tablet 2344a170834645609b6a0774aa79a709 (DEFAULT_TABLE table=test-table [id=80a39bc6485940d1affd50634eee5efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:36:30.154071 10886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344a170834645609b6a0774aa79a709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:30.154170 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344a170834645609b6a0774aa79a709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:30.154286 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2344a170834645609b6a0774aa79a709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:36:30.161091 11103 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Bootstrap starting.
I20260620 04:36:30.161939 11105 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Bootstrap starting.
I20260620 04:36:30.163311 11103 tablet_bootstrap.cc:654] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:30.163765 11104 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: Bootstrap starting.
I20260620 04:36:30.163938 11105 tablet_bootstrap.cc:654] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:30.164683 11103 log.cc:824] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:30.164752 11105 log.cc:824] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:30.166613 11105 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: No bootstrap required, opened a new log
I20260620 04:36:30.166750 11103 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: No bootstrap required, opened a new log
I20260620 04:36:30.166800 11104 tablet_bootstrap.cc:654] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: Neither blocks nor log segments found. Creating new log.
I20260620 04:36:30.166831 11105 ts_tablet_manager.cc:1397] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:36:30.166939 11103 ts_tablet_manager.cc:1397] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:36:30.167608 11104 log.cc:824] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:36:30.169634 11104 tablet_bootstrap.cc:492] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: No bootstrap required, opened a new log
I20260620 04:36:30.169824 11104 ts_tablet_manager.cc:1397] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:36:30.170207 11105 raft_consensus.cc:348] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.170513 11105 raft_consensus.cc:374] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:30.170576 11105 raft_consensus.cc:729] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d6ff9ac9a64f118aebad4a2ab1a199, State: Initialized, Role: FOLLOWER
I20260620 04:36:30.170766 11103 raft_consensus.cc:348] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.171013 11103 raft_consensus.cc:374] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:30.171074 11103 raft_consensus.cc:729] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca97aa2f6c48413a8dee2cabcf075e72, State: Initialized, Role: FOLLOWER
I20260620 04:36:30.171060 11105 consensus_queue.cc:260] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.171571 11103 consensus_queue.cc:260] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.172036 11105 ts_tablet_manager.cc:1428] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:36:30.172420 11086 heartbeater.cc:503] Master 127.0.98.62:46069 was elected leader, sending a full tablet report...
I20260620 04:36:30.172420 10952 heartbeater.cc:503] Master 127.0.98.62:46069 was elected leader, sending a full tablet report...
I20260620 04:36:30.172642 11103 ts_tablet_manager.cc:1428] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 04:36:30.174451 11104 raft_consensus.cc:348] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.174726 11104 raft_consensus.cc:374] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:36:30.174787 11104 raft_consensus.cc:729] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fef56473f543bf8fe1c27f0833732c, State: Initialized, Role: FOLLOWER
I20260620 04:36:30.175249 11104 consensus_queue.cc:260] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.176141 11104 ts_tablet_manager.cc:1428] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:36:30.176365 10818 heartbeater.cc:503] Master 127.0.98.62:46069 was elected leader, sending a full tablet report...
I20260620 04:36:30.298782 11109 raft_consensus.cc:482] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:36:30.299053 11109 raft_consensus.cc:504] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.300341 11109 leader_election.cc:288] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca97aa2f6c48413a8dee2cabcf075e72 (127.0.98.3:39827), a5fef56473f543bf8fe1c27f0833732c (127.0.98.1:37515)
I20260620 04:36:30.304452 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.301134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47745 (local address 127.0.98.1:37515)
0620 04:36:30.301261 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.301265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.301279 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:36:30.301385 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.301391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.301449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.301517 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:30.301836 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.302304 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.303455 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.303863 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.303976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.304051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.304247 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.304322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:36:30.304457 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.300910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37515 (local address 127.0.98.2:47745)
0620 04:36:30.301199 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.301212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.301296 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.301548 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.301551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.301567 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:30.301732 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.301738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.302406 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.302410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.303329 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.303338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.303987 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.303990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.304090 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.304112 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.304182 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.304303 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":35,"thread_start_us":130,"threads_started":1}
I20260620 04:36:30.304735 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.300737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39723 (local address 127.0.98.3:39827)
0620 04:36:30.300868 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.300872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.300888 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:36:30.301014 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.301019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.301075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.301158 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:36:30.301464 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.302309 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.303460 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.303875 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.303993 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.304052 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.304511 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.304603 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:36:30.304463 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.300636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39827 (local address 127.0.98.2:39723)
0620 04:36:30.300764 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.300781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:36:30.300871 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:36:30.301203 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:36:30.301206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:36:30.301224 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:36:30.301348 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.301356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.302445 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.302448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.303331 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:36:30.303340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.304018 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.304022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:36:30.304145 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.304181 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:36:30.304272 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 04:36:30.304338 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":82}
I20260620 04:36:30.305212 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344a170834645609b6a0774aa79a709" candidate_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fef56473f543bf8fe1c27f0833732c" is_pre_election: true
I20260620 04:36:30.305351 11040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344a170834645609b6a0774aa79a709" candidate_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" is_pre_election: true
I20260620 04:36:30.305553 10772 raft_consensus.cc:2454] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1d6ff9ac9a64f118aebad4a2ab1a199 in term 0.
I20260620 04:36:30.305698 11040 raft_consensus.cc:2454] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1d6ff9ac9a64f118aebad4a2ab1a199 in term 0.
I20260620 04:36:30.306056 10842 leader_election.cc:302] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d6ff9ac9a64f118aebad4a2ab1a199, a5fef56473f543bf8fe1c27f0833732c; no voters:
I20260620 04:36:30.306317 11109 raft_consensus.cc:2781] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:36:30.306401 11109 raft_consensus.cc:482] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:36:30.306445 11109 raft_consensus.cc:3037] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:30.307448 11109 raft_consensus.cc:504] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.307781 11109 leader_election.cc:288] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [CANDIDATE]: Term 1 election: Requested vote from peers ca97aa2f6c48413a8dee2cabcf075e72 (127.0.98.3:39827), a5fef56473f543bf8fe1c27f0833732c (127.0.98.1:37515)
I20260620 04:36:30.308148 11040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344a170834645609b6a0774aa79a709" candidate_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca97aa2f6c48413a8dee2cabcf075e72"
I20260620 04:36:30.308223 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2344a170834645609b6a0774aa79a709" candidate_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fef56473f543bf8fe1c27f0833732c"
I20260620 04:36:30.308337 10772 raft_consensus.cc:3037] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:30.308339 11040 raft_consensus.cc:3037] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:36:30.309322 10772 raft_consensus.cc:2454] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1d6ff9ac9a64f118aebad4a2ab1a199 in term 1.
I20260620 04:36:30.309352 11040 raft_consensus.cc:2454] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1d6ff9ac9a64f118aebad4a2ab1a199 in term 1.
I20260620 04:36:30.309592 10842 leader_election.cc:302] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d6ff9ac9a64f118aebad4a2ab1a199, a5fef56473f543bf8fe1c27f0833732c; no voters:
I20260620 04:36:30.309803 11109 raft_consensus.cc:2781] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:36:30.310047 11109 raft_consensus.cc:686] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 1 LEADER]: Becoming Leader. State: Replica: a1d6ff9ac9a64f118aebad4a2ab1a199, State: Running, Role: LEADER
I20260620 04:36:30.310403 11109 consensus_queue.cc:237] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } }
I20260620 04:36:30.313807 10627 catalog_manager.cc:5495] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1d6ff9ac9a64f118aebad4a2ab1a199 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1d6ff9ac9a64f118aebad4a2ab1a199" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34207 } health_report { overall_health: HEALTHY } } }
W20260620 04:36:30.341814 11087 tablet.cc:2369] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:30.345402 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.341572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.0.98.2:34207)
0620 04:36:30.341759 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 04:36:30.341766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:36:30.341783 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:36:30.341876 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:36:30.341878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:36:30.341964 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:36:30.342068 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:36:30.342309 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.342934 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.343832 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:36:30.344076 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:36:30.344193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:36:30.344551 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:36:30.344966 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:36:30.345107 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:36:30.345179 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:36:30.345235 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 04:36:30.351135 10772 raft_consensus.cc:1261] T 2344a170834645609b6a0774aa79a709 P a5fef56473f543bf8fe1c27f0833732c [term 1 FOLLOWER]: Refusing update from remote peer a1d6ff9ac9a64f118aebad4a2ab1a199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:30.351186 11040 raft_consensus.cc:1261] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 1 FOLLOWER]: Refusing update from remote peer a1d6ff9ac9a64f118aebad4a2ab1a199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:36:30.352006 11113 consensus_queue.cc:1035] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fef56473f543bf8fe1c27f0833732c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:36:30.352229 11109 consensus_queue.cc:1035] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca97aa2f6c48413a8dee2cabcf075e72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:36:30.356436 10953 tablet.cc:2369] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:36:30.366236 11118 mvcc.cc:205] Tried to move back new op lower bound from 7298786059671298048 to 7298786059513864192. Current Snapshot: MvccSnapshot[applied={T|T < 7298786059671298048}]
I20260620 04:36:30.368661 11117 mvcc.cc:205] Tried to move back new op lower bound from 7298786059671298048 to 7298786059513864192. Current Snapshot: MvccSnapshot[applied={T|T < 7298786059671298048}]
I20260620 04:36:30.370507 10625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:30.370697 10625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:36:30.373142 10625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b23376d96584ef085303abbba27a8cc: Sending DeleteTablet for 3 replicas of tablet 2344a170834645609b6a0774aa79a709
I20260620 04:36:30.374666 10886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344a170834645609b6a0774aa79a709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46230
I20260620 04:36:30.374675 10752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344a170834645609b6a0774aa79a709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35110
I20260620 04:36:30.375339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10689
I20260620 04:36:30.375375 11020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2344a170834645609b6a0774aa79a709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:36:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38060
I20260620 04:36:30.375391 11126 tablet_replica.cc:330] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: stopping tablet replica
I20260620 04:36:30.376389 11126 raft_consensus.cc:2229] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:36:30.376895 11126 raft_consensus.cc:2258] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:30.377928 11128 tablet_replica.cc:330] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: stopping tablet replica
I20260620 04:36:30.378428 11128 raft_consensus.cc:2229] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:36:30.378856 11128 raft_consensus.cc:2258] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:36:30.379832 11126 ts_tablet_manager.cc:1905] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:30.380396 11128 ts_tablet_manager.cc:1905] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:36:30.383778 11126 ts_tablet_manager.cc:1918] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:30.383968 11126 log.cc:1196] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/2344a170834645609b6a0774aa79a709
I20260620 04:36:30.384374 11126 ts_tablet_manager.cc:1939] T 2344a170834645609b6a0774aa79a709 P a1d6ff9ac9a64f118aebad4a2ab1a199: Deleting consensus metadata
I20260620 04:36:30.385203 11128 ts_tablet_manager.cc:1918] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:36:30.385311 11128 log.cc:1196] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/2344a170834645609b6a0774aa79a709
I20260620 04:36:30.385643 11128 ts_tablet_manager.cc:1939] T 2344a170834645609b6a0774aa79a709 P ca97aa2f6c48413a8dee2cabcf075e72: Deleting consensus metadata
I20260620 04:36:30.385891 10611 catalog_manager.cc:4841] TS a1d6ff9ac9a64f118aebad4a2ab1a199 (127.0.98.2:34207): tablet 2344a170834645609b6a0774aa79a709 (table test-table [id=80a39bc6485940d1affd50634eee5efb]) successfully deleted
W20260620 04:36:30.387070 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37515 (error 108)
I20260620 04:36:30.387233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10823
I20260620 04:36:30.387523 10614 catalog_manager.cc:4841] TS ca97aa2f6c48413a8dee2cabcf075e72 (127.0.98.3:39827): tablet 2344a170834645609b6a0774aa79a709 (table test-table [id=80a39bc6485940d1affd50634eee5efb]) successfully deleted
I20260620 04:36:30.388877 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:36:30.388589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37515 (local address 127.0.0.1:35112)
0620 04:36:30.388720 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:36:30.388794 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:36:30.389024 10614 catalog_manager.cc:4584] TS a5fef56473f543bf8fe1c27f0833732c (127.0.98.1:37515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2344a170834645609b6a0774aa79a709: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37515: connect: Connection refused (error 111)
I20260620 04:36:30.392758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10957
I20260620 04:36:30.397743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 10595
2026-06-20T04:36:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2452 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:36:30 dist-test-slave-zmf4 krb5kdc[11132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:36:30 dist-test-slave-zmf4 krb5kdc[11132](info): set up 2 sockets
Jun 20 04:36:30 dist-test-slave-zmf4 krb5kdc[11132](info): commencing operation
krb5kdc: starting...
W20260620 04:36:31.461290 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:36:31 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:36:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:36:31.635032 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGAOikI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:36:34.363 UTC [11162] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:36:34.363 UTC [11162] LOG: listening on IPv4 address "127.0.98.20", port 51171
2026-06-20 04:36:34.368 UTC [11162] LOG: listening on Unix socket "/tmp/.s.PGSQL.51171"
2026-06-20 04:36:34.390 UTC [11165] LOG: database system was shut down at 2026-06-20 04:36:32 UTC
2026-06-20 04:36:34.394 UTC [11162] LOG: database system is ready to accept connections
I20260620 04:36:35.373512 392 mini_postgres.cc:96] Postgres bound to 51171
2026-06-20 04:36:35.379 UTC [11173] FATAL: database "slave" does not exist
127.0.98.20:51171 - accepting connections
I20260620 04:36:35.380573 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:36:35.395982 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:36:35.940424 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:36:35.940543 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin
2026-06-20 04:36:36,234 [I] DB FLAVOR :POSTGRES
2026-06-20 04:36:36,235 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:36:36,235 [I] Checking connection..
2026-06-20 04:36:36,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:36:36,509 [I] Checking connection passed.
2026-06-20 04:36:36,509 [I] --------- Verifying version history table ---------
2026-06-20 04:36:36,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:36:36,773 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:36:36,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:36:37,040 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:36:37,040 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:36:37,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:36:37.316 UTC [11237] WARNING: there is no transaction in progress
2026-06-20 04:36:37,324 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:36:37,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:36:37,597 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:36:37,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:36:37,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:36:37,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:36:38,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zmf4.c.gcp-upstream.internal','N') ;"
2026-06-20 04:36:38,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:36:38,647 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:36:38,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:36:38,906 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:36:38,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:36:39,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:36:39,435 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:36:39,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:36:40.488 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.504 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.523 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.540 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.561 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.582 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.644 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.652 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.668 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.679 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.691 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.707 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.719 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.727 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.737 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:40.746 UTC [11345] WARNING: there is no transaction in progress
2026-06-20 04:36:41,508 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:36:41,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:36:41,771 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:36:41,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:36:42,031 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:36:42,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:36:42,306 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:36:42,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:36:42,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:36:42,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zmf4.c.gcp-upstream.internal';"
2026-06-20 04:36:43,043 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:36:43,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:36:43,312 [I] DB_PATCHES have already been applied
I20260620 04:36:43.318213 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGAOikI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:36:43.318316 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:36:43.321784 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52277
Jun 20, 2026 4:36:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:36:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:36:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:36:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:36:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:36:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:36:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:36:44.480576 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.100s user 0.003s sys 0.008s
Jun 20, 2026 4:37:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:37:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 4:37:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:37:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:37:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:37:07.327545 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:37:07 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930227, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:37:07.415114 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:37:07.425033 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:37:08.493657 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:37:08.493746 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms
I20260620 04:37:08.493842 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51171
2026-06-20 04:37:08,702 [I] DB FLAVOR :POSTGRES
2026-06-20 04:37:08,702 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:37:08,702 [I] Checking connection
2026-06-20 04:37:08,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:37:08,978 [I] connection success
2026-06-20 04:37:08,978 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:37:08,978 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:37:08,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:37:09,242 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:37:09,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:37:09,242 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:37:09,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:37:09,532 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:37:10.066740 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:37:10.266837 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:37:10.334751 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:37:10 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930227, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:10.771446 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:37:10.771597 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:37:10.771618 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:37:10.775429 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260620 04:37:10.775561 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34315
Jun 20, 2026 4:37:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:37:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:37:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:37:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:37:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:37:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 04:37:13 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781930233, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:37:13.111291 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.696s user 0.000s sys 0.192s
I20260620 04:37:13.111423 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:37:13.111534 392 mini_ranger_kms.cc:401] 127.0.98.20:34315/kms/v1/keys
Jun 20 04:37:13 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930233, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:37:13 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930233, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:37:16 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:37:16.232272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38857
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34315/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:37:16.355661 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:37:16.355965 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:37:16.356035 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:37:16.360553 11576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:37:16.360647 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:37:16.360803 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:37:16.360848 11576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:37:16.360932 11576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:37:16.367051 11576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34315/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38857
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:37:16.368073 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:37:16.369244 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:37:16.378213 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:16.378458 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:37:16.378454 11576 server_base.cc:1034] running on GCE node
W20260620 04:37:16.378542 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:37:16.379187 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:37:16.380551 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:37:16.381812 11576 hybrid_clock.cc:648] HybridClock initialized: now 1781930236381778 us; error 46 us; skew 500 ppm
Jun 20 04:37:16 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:37:16.388976 11576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:37:16.389992 11576 webserver.cc:460] Webserver started at http://127.0.98.62:44223/ using document root <none> and password file <none>
I20260620 04:37:16.390520 11576 fs_manager.cc:359] Metadata directory not provided
I20260620 04:37:16.390573 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:37:16.390784 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:37:16 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:16.436949 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "8f253a2b57d846d08f2c6f13db265044"
format_stamp: "Formatted at 2026-06-20 04:37:16 on dist-test-slave-zmf4"
server_key: "041c3de9dddcd6186bb2c04ed38cbcbe"
server_key_iv: "dda99e55d15242a12fe0e55f53c753c8"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:16.437577 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f253a2b57d846d08f2c6f13db265044"
format_stamp: "Formatted at 2026-06-20 04:37:16 on dist-test-slave-zmf4"
server_key: "041c3de9dddcd6186bb2c04ed38cbcbe"
server_key_iv: "dda99e55d15242a12fe0e55f53c753c8"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:16.441529 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:37:16.455811 11591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:16.457226 11576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:37:16.457372 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "8f253a2b57d846d08f2c6f13db265044"
format_stamp: "Formatted at 2026-06-20 04:37:16 on dist-test-slave-zmf4"
server_key: "041c3de9dddcd6186bb2c04ed38cbcbe"
server_key_iv: "dda99e55d15242a12fe0e55f53c753c8"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:16.457520 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:37:16.470256 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:37:16.474853 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:37:16.475023 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:37:16.482940 11576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38857
I20260620 04:37:16.482964 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38857 every 8 connection(s)
I20260620 04:37:16.483896 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:37:16.487141 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:37:16.490223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 11576
I20260620 04:37:16.490384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:37:16.490686 392 external_mini_cluster.cc:1428] Setting key 2e3617c3f7f6fc324198ea64f9a69694
I20260620 04:37:16.494614 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: Bootstrap starting.
I20260620 04:37:16.497282 11644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: Neither blocks nor log segments found. Creating new log.
I20260620 04:37:16.498219 11644 log.cc:824] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:37:16 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:37:16.500305 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: No bootstrap required, opened a new log
I20260620 04:37:16.504048 11644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260620 04:37:16.504334 11644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:37:16.504407 11644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f253a2b57d846d08f2c6f13db265044, State: Initialized, Role: FOLLOWER
I20260620 04:37:16.504894 11644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260620 04:37:16.505080 11644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:37:16.505182 11644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:37:16.505299 11644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:37:16.506369 11644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260620 04:37:16.506417 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:16.492090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.98.62:38857)
0620 04:37:16.492550 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:16.492574 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:16.492609 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:37:16.493249 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:16.493265 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:16.493583 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:16.493920 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:37:16.494670 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:16.495678 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:16.496586 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:16.497057 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:16.497222 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:16.499904 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:37:16.499910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:37:16.499925 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:37:16.499957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:37:16.502883 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:16.503407 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:16.503412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:16.503418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:16.503550 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:16.503859 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:16.503862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:16.503864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:16.504374 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:37:16.504572 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:16.504967 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 04:37:16.505497 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260620 04:37:16.506728 11644 leader_election.cc:302] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f253a2b57d846d08f2c6f13db265044; no voters:
I20260620 04:37:16.507110 11644 leader_election.cc:288] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:37:16.507825 11649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:37:16.508109 11649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [term 1 LEADER]: Becoming Leader. State: Replica: 8f253a2b57d846d08f2c6f13db265044, State: Running, Role: LEADER
I20260620 04:37:16.508476 11649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260620 04:37:16.509037 11644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:37:16.509821 11649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f253a2b57d846d08f2c6f13db265044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } } }
I20260620 04:37:16.510025 11649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [sys.catalog]: This master's current role is: LEADER
I20260620 04:37:16.509867 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f253a2b57d846d08f2c6f13db265044. Latest consensus state: current_term: 1 leader_uuid: "8f253a2b57d846d08f2c6f13db265044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f253a2b57d846d08f2c6f13db265044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } } }
I20260620 04:37:16.510314 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044 [sys.catalog]: This master's current role is: LEADER
I20260620 04:37:16.510567 11657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:37:16.513523 11657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:37:16.513952 11644 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:37:16.523581 11666 mvcc.cc:205] Tried to move back new op lower bound from 7298786248784044032 to 7298786248739696640. Current Snapshot: MvccSnapshot[applied={T|T < 7298786248784044032}]
I20260620 04:37:16.525879 11657 catalog_manager.cc:1347] Generated new cluster ID: f3c63e84942e4a1d947260fe4c21e5d3
I20260620 04:37:16.525962 11657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:37:16.553632 11657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:37:16.554898 11657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:37:16.561558 11657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: Generated new TSK 0
I20260620 04:37:16.562356 11657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:37:17.343392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:37:17.495213 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:37:17.495580 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:37:17.495643 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:37:17.502125 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:37:17.502229 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:37:17.502338 11689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:37:17.508013 11689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:37:17.509344 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:37:17.510722 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:37:17.520555 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:17.520501 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:17.520784 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:37:17.521890 11689 server_base.cc:1034] running on GCE node
I20260620 04:37:17.522327 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:37:17.523072 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:37:17.524302 11689 hybrid_clock.cc:648] HybridClock initialized: now 1781930237524271 us; error 48 us; skew 500 ppm
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:37:17.527747 11689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:37:17.528944 11689 webserver.cc:460] Webserver started at http://127.0.98.1:35733/ using document root <none> and password file <none>
I20260620 04:37:17.529703 11689 fs_manager.cc:359] Metadata directory not provided
I20260620 04:37:17.529806 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:37:17.530056 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:17.551307 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "20a55204745e4f12bca2662d151dca1d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "c228912fbd5b507ff1c98593dba81e2d"
server_key_iv: "2597f710ba090497c48c1812772fa0c9"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.552181 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20a55204745e4f12bca2662d151dca1d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "c228912fbd5b507ff1c98593dba81e2d"
server_key_iv: "2597f710ba090497c48c1812772fa0c9"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.557754 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 04:37:17.584836 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.586525 11689 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 04:37:17.586652 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "20a55204745e4f12bca2662d151dca1d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "c228912fbd5b507ff1c98593dba81e2d"
server_key_iv: "2597f710ba090497c48c1812772fa0c9"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.586782 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:37:17.614235 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:37:17.618101 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:37:17.618366 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:37:17.619513 11689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:37:17.620675 11689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:37:17.620786 11689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.620939 11689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:37:17.621026 11689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.631711 11689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41109
I20260620 04:37:17.632993 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:37:17.640935 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41109 every 8 connection(s)
I20260620 04:37:17.642385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 11689
I20260620 04:37:17.642529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:37:17.642793 392 external_mini_cluster.cc:1428] Setting key e802bb0597717a55dbe3afb9f1823407
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:37:17.657662 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:17.641344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58349 (local address 127.0.98.62:38857)
0620 04:37:17.641660 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:17.641664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:17.642063 (+ 399us) server_negotiation.cc:407] Connection header received
0620 04:37:17.643321 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:17.643326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:17.643382 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:17.643443 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:37:17.646680 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.647534 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:17.649403 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.649736 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:17.649822 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:17.653074 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:37:17.653078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:37:17.653081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:37:17.653110 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:37:17.655052 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:17.655691 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:17.655695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:17.655697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:17.655800 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:17.656287 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:17.656290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:17.656292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:17.656527 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:37:17.656623 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:17.657339 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 04:37:17.657474 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":81,"threads_started":1}
I20260620 04:37:17.659631 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:17.641221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38857 (local address 127.0.98.1:58349)
0620 04:37:17.641913 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:17.641966 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:17.643062 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:17.643849 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:17.643858 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:17.645159 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:37:17.646383 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:17.646429 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.647727 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:17.647732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:17.649239 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:17.649253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.650074 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:17.650078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:17.650283 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:17.651012 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:37:17.651035 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:37:17.652841 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:37:17.655183 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:17.655190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:17.655209 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:17.655552 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:17.655933 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:17.655937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:17.655939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:17.656132 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:17.656676 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:37:17.656683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:37:17.657057 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 04:37:17.658246 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0620 04:37:17.658593 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":138,"threads_started":1}
I20260620 04:37:17.661429 11822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38857
I20260620 04:37:17.661677 11822 heartbeater.cc:464] Registering TS with master...
I20260620 04:37:17.662308 11822 heartbeater.cc:511] Master 127.0.98.62:38857 requested a full tablet report, sending...
I20260620 04:37:17.664151 11608 ts_manager.cc:194] Registered new tserver with Master: 20a55204745e4f12bca2662d151dca1d (127.0.98.1:41109)
I20260620 04:37:17.665725 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:37:17.676023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:37:17.835925 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:37:17.836344 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:37:17.836416 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:37:17.843246 11828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:37:17.843343 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:37:17.843436 11828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:37:17.851666 11828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:37:17.852988 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:37:17.854179 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:37:17.864831 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:17.865134 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:17.865264 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:37:17.865859 11828 server_base.cc:1034] running on GCE node
I20260620 04:37:17.866285 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:37:17.866950 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:37:17.868145 11828 hybrid_clock.cc:648] HybridClock initialized: now 1781930237868110 us; error 47 us; skew 500 ppm
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:37:17.871073 11828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:37:17.872026 11828 webserver.cc:460] Webserver started at http://127.0.98.2:38153/ using document root <none> and password file <none>
I20260620 04:37:17.872646 11828 fs_manager.cc:359] Metadata directory not provided
I20260620 04:37:17.872735 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:37:17.872962 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:17.888036 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "117497b3855a4405b9879b8194a2005d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "4ae4e9bb0c2010dd1ec69cd29472c9ab"
server_key_iv: "c8764669c1204edc6d38fb71894911e0"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.888811 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "117497b3855a4405b9879b8194a2005d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "4ae4e9bb0c2010dd1ec69cd29472c9ab"
server_key_iv: "c8764669c1204edc6d38fb71894911e0"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.894073 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 04:37:17.924690 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.934320 11828 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.001s sys 0.000s
I20260620 04:37:17.934520 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "117497b3855a4405b9879b8194a2005d"
format_stamp: "Formatted at 2026-06-20 04:37:17 on dist-test-slave-zmf4"
server_key: "4ae4e9bb0c2010dd1ec69cd29472c9ab"
server_key_iv: "c8764669c1204edc6d38fb71894911e0"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:17.934693 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:37:17.965152 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:37:17.969372 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:37:17.969569 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:37:17.970292 11828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:37:17.971272 11828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:37:17.971339 11828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.971411 11828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:37:17.971444 11828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:17.983215 11828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40041
I20260620 04:37:17.984385 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:37:17.988422 11957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40041 every 8 connection(s)
I20260620 04:37:17.994274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 11828
I20260620 04:37:17.994415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:37:17.994738 392 external_mini_cluster.cc:1428] Setting key 60cec391260a3af734ecb6f8be58e381
Jun 20 04:37:17 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:37:18.009749 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:17.990501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49365 (local address 127.0.98.62:38857)
0620 04:37:17.990651 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:17.990656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:17.992566 (+ 1910us) server_negotiation.cc:407] Connection header received
0620 04:37:17.993885 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:17.993889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:17.993972 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:17.994030 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:37:17.996959 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.997787 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.000303 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.000671 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.000754 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.005198 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:37:18.005203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:37:18.005206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:37:18.005238 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:37:18.007104 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.007761 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.007765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.007767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.007856 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.008306 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.008309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.008311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.008537 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:37:18.008623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.009407 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.009551 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:37:18.011658 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:17.990984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38857 (local address 127.0.98.2:49365)
0620 04:37:17.992393 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:17.992454 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:17.993623 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:17.994457 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:17.994467 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:17.994825 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:37:17.996780 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:17.996803 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:17.997954 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:17.997957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:17.999358 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:17.999369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.000966 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.000971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.001172 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.002581 (+ 1409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:37:18.002604 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:37:18.004328 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:37:18.007253 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:18.007259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:18.007277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:18.007639 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:18.007965 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:18.007968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:18.007970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:18.008148 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:18.008692 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:37:18.008699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:37:18.009094 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.010113 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.010509 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:37:18.013216 11958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38857
I20260620 04:37:18.013464 11958 heartbeater.cc:464] Registering TS with master...
I20260620 04:37:18.014014 11958 heartbeater.cc:511] Master 127.0.98.62:38857 requested a full tablet report, sending...
I20260620 04:37:18.015072 11608 ts_manager.cc:194] Registered new tserver with Master: 117497b3855a4405b9879b8194a2005d (127.0.98.2:40041)
I20260620 04:37:18.015825 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:37:18.031358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:37:18 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 04:37:18.205134 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:37:18.205547 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:37:18.205607 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:37:18.211663 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:37:18.211747 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:37:18.211841 11963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:37:18.219966 11963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:37:18.221359 11963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:37:18.222602 11963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:37:18.234792 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:18.234881 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:37:18.234951 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:37:18.235352 11963 server_base.cc:1034] running on GCE node
I20260620 04:37:18.235826 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:37:18.236644 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:37:18.237937 11963 hybrid_clock.cc:648] HybridClock initialized: now 1781930238237866 us; error 85 us; skew 500 ppm
Jun 20 04:37:18 dist-test-slave-zmf4 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:37:18.241609 11963 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:37:18.242542 11963 webserver.cc:460] Webserver started at http://127.0.98.3:35575/ using document root <none> and password file <none>
I20260620 04:37:18.243124 11963 fs_manager.cc:359] Metadata directory not provided
I20260620 04:37:18.243216 11963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:37:18.243444 11963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:37:18 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:18.260919 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "a20f48ea33d6404a838384dec79d5eb2"
format_stamp: "Formatted at 2026-06-20 04:37:18 on dist-test-slave-zmf4"
server_key: "92636d4b2ec5b321c55d294281933b36"
server_key_iv: "d6b7f06f34d5d46b3ed5011f664eeabc"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:18.261704 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a20f48ea33d6404a838384dec79d5eb2"
format_stamp: "Formatted at 2026-06-20 04:37:18 on dist-test-slave-zmf4"
server_key: "92636d4b2ec5b321c55d294281933b36"
server_key_iv: "d6b7f06f34d5d46b3ed5011f664eeabc"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:18.267634 11963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 04:37:18.300714 11978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:18.302664 11963 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260620 04:37:18.302837 11963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "a20f48ea33d6404a838384dec79d5eb2"
format_stamp: "Formatted at 2026-06-20 04:37:18 on dist-test-slave-zmf4"
server_key: "92636d4b2ec5b321c55d294281933b36"
server_key_iv: "d6b7f06f34d5d46b3ed5011f664eeabc"
server_key_version: "kuduclusterkey@0"
I20260620 04:37:18.303004 11963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:37:18.317935 11963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:37:18.321686 11963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:37:18.321898 11963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:37:18.322551 11963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:37:18.323572 11963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:37:18.323637 11963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:18.323704 11963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:37:18.323737 11963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:37:18.345731 11963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38927
I20260620 04:37:18.346370 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38927 every 8 connection(s)
I20260620 04:37:18.347100 11963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:37:18.351420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 11963
I20260620 04:37:18.351567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:37:18.351836 392 external_mini_cluster.cc:1428] Setting key b849476104ef990bef770368abb9111c
Jun 20 04:37:18 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:37:18.364037 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.349151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48355 (local address 127.0.98.62:38857)
0620 04:37:18.349282 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.349287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.350204 (+ 917us) server_negotiation.cc:407] Connection header received
0620 04:37:18.351210 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.351216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.351279 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.351345 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:37:18.353922 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.354479 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.355646 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.355873 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.355987 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.359658 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:37:18.359662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:37:18.359665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:37:18.359696 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:37:18.361496 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.362188 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.362192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.362194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.362302 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.362734 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.362741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.362742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.362963 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:37:18.363035 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.363746 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.363870 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:37:18.366097 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.349582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38857 (local address 127.0.98.3:48355)
0620 04:37:18.350055 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.350106 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.350959 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.352508 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.352518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.353087 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:37:18.353749 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.353770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.354659 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.354662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.355520 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.355528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.356024 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.356027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.356251 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.356886 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:37:18.357006 (+ 120us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:37:18.359447 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:37:18.361681 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:18.361687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:18.361704 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:18.362053 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:18.362424 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:37:18.362428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:37:18.362430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:37:18.362613 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:37:18.363122 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:37:18.363128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:37:18.363471 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.364525 (+ 1054us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.364881 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":142,"threads_started":1}
I20260620 04:37:18.367133 12092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38857
I20260620 04:37:18.367372 12092 heartbeater.cc:464] Registering TS with master...
I20260620 04:37:18.367857 12092 heartbeater.cc:511] Master 127.0.98.62:38857 requested a full tablet report, sending...
I20260620 04:37:18.368911 11608 ts_manager.cc:194] Registered new tserver with Master: a20f48ea33d6404a838384dec79d5eb2 (127.0.98.3:38927)
I20260620 04:37:18.369467 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48355
I20260620 04:37:18.377909 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:37:18.387581 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.380024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.62:38857)
0620 04:37:18.380142 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.380147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.380394 (+ 247us) server_negotiation.cc:407] Connection header received
0620 04:37:18.380529 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.380532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.380577 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.380670 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:37:18.381365 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.381888 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.382710 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.382948 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.383051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.384496 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:37:18.384500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:37:18.384503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:37:18.384527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:37:18.385966 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.386384 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.386389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.386391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.386467 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:37:18.386740 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:37:18.386743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:37:18.386745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:37:18.386925 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:37:18.387063 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.387294 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.387402 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:37:18.390777 11608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:37:18.395562 11608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:37:18.427444 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.415050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38927 (local address 127.0.0.1:39118)
0620 04:37:18.415949 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.416003 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.416135 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.417018 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.417024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.421368 (+ 4344us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:37:18.421539 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.421548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.423231 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.423234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.425150 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.425161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426922 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.426926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.427036 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.427079 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.427176 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.427285 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260620 04:37:18.427940 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.416064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.98.3:38927)
0620 04:37:18.416476 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.416491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.416516 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:37:18.416593 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.416599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.416834 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.421723 (+ 4889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:37:18.421867 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.423081 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.425651 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426598 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.426696 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.426846 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.427673 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.427787 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
I20260620 04:37:18.430362 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.418469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40041 (local address 127.0.0.1:37652)
0620 04:37:18.424279 (+ 5810us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.424294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.424391 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.425010 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.425013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.425029 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:37:18.425111 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.425116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426523 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.426527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.427780 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.427790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.429626 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.429629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.430056 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.430080 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.430148 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.430216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5746,"mutex_wait_us":321,"thread_start_us":85,"threads_started":1}
I20260620 04:37:18.431387 12026 tablet_service.cc:1467] Processing CreateTablet for tablet a40aea7abc4b42df95b6141569e3c8ef (DEFAULT_TABLE table=test-table [id=b8632f99dda2492487a74fce9f356abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:37:18.432597 12026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40aea7abc4b42df95b6141569e3c8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:37:18.434131 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.420778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.98.2:40041)
0620 04:37:18.424112 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.424129 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.424489 (+ 360us) server_negotiation.cc:407] Connection header received
0620 04:37:18.424567 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.424572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.424740 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.424875 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:37:18.425232 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426388 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.427938 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.429307 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.429422 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.429563 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.431557 (+ 1994us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.433983 (+ 2426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3231,"thread_start_us":107,"threads_started":1}
I20260620 04:37:18.437810 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.417507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41109 (local address 127.0.0.1:40954)
0620 04:37:18.420385 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.420401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.421782 (+ 1381us) spinlock_profiling.cc:243] Waited 918 us on lock 0x1772d028. stack: 00007f71c18e2c78 00007f71c18e31a9 00007f71c14a51d7 00000000005984a6 00000000005984f9 000000000059c488 00007f71c18df2ab 00007f71c18de9da 00007f71ca03c817 00007f71ca04f07f 00007f71ca045003 00007f71c5f37267 00007f71c5f35819 00007f71c5f76832 00007f71c5f7768c 00007f71c5f8d0d3
0620 04:37:18.421806 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.424496 (+ 2690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.424499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.424518 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:37:18.424636 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.424642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426599 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.426603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.429771 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.429779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.437067 (+ 7288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.437071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.437194 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.437222 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.437283 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.437652 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2800,"mutex_wait_us":4,"spinlock_wait_cycles":2111360,"thread_start_us":93,"threads_started":1}
I20260620 04:37:18.441475 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.420318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.98.1:41109)
0620 04:37:18.423371 (+ 3053us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.423379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.423419 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:37:18.423496 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.423502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.423708 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.423850 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:37:18.424777 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.426437 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.430627 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.436407 (+ 5780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.437893 (+ 1486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.438062 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.438135 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.438234 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2938,"thread_start_us":97,"threads_started":1}
I20260620 04:37:18.443985 11892 tablet_service.cc:1467] Processing CreateTablet for tablet a40aea7abc4b42df95b6141569e3c8ef (DEFAULT_TABLE table=test-table [id=b8632f99dda2492487a74fce9f356abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:37:18.445513 12111 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Bootstrap starting.
I20260620 04:37:18.446013 11892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40aea7abc4b42df95b6141569e3c8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:37:18.447278 11756 tablet_service.cc:1467] Processing CreateTablet for tablet a40aea7abc4b42df95b6141569e3c8ef (DEFAULT_TABLE table=test-table [id=b8632f99dda2492487a74fce9f356abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:37:18.448237 12111 tablet_bootstrap.cc:654] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Neither blocks nor log segments found. Creating new log.
I20260620 04:37:18.448289 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40aea7abc4b42df95b6141569e3c8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:37:18.449102 12111 log.cc:824] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:37:18.451191 12111 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: No bootstrap required, opened a new log
I20260620 04:37:18.451400 12111 ts_tablet_manager.cc:1397] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:37:18.455720 12111 raft_consensus.cc:348] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.455997 12111 raft_consensus.cc:374] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:37:18.456046 12111 raft_consensus.cc:729] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20f48ea33d6404a838384dec79d5eb2, State: Initialized, Role: FOLLOWER
I20260620 04:37:18.456547 12111 consensus_queue.cc:260] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.458084 12111 ts_tablet_manager.cc:1428] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 04:37:18.458498 12092 heartbeater.cc:503] Master 127.0.98.62:38857 was elected leader, sending a full tablet report...
I20260620 04:37:18.458823 12113 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Bootstrap starting.
I20260620 04:37:18.462287 12115 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Bootstrap starting.
Jun 20 04:37:18 dist-test-slave-zmf4 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:18.465219 12115 tablet_bootstrap.cc:654] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Neither blocks nor log segments found. Creating new log.
I20260620 04:37:18.465220 12113 tablet_bootstrap.cc:654] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Neither blocks nor log segments found. Creating new log.
I20260620 04:37:18.466114 12115 log.cc:824] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:37:18.466404 12113 log.cc:824] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:37:18.468317 12115 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: No bootstrap required, opened a new log
I20260620 04:37:18.468511 12115 ts_tablet_manager.cc:1397] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 04:37:18.474192 12115 raft_consensus.cc:348] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.474526 12115 raft_consensus.cc:374] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:37:18.474581 12115 raft_consensus.cc:729] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a55204745e4f12bca2662d151dca1d, State: Initialized, Role: FOLLOWER
I20260620 04:37:18.475095 12115 consensus_queue.cc:260] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.476277 12115 ts_tablet_manager.cc:1428] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260620 04:37:18.476846 11822 heartbeater.cc:503] Master 127.0.98.62:38857 was elected leader, sending a full tablet report...
I20260620 04:37:18.479575 12113 tablet_bootstrap.cc:492] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: No bootstrap required, opened a new log
I20260620 04:37:18.479797 12113 ts_tablet_manager.cc:1397] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260620 04:37:18.484028 12113 raft_consensus.cc:348] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.484354 12113 raft_consensus.cc:374] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:37:18.484401 12113 raft_consensus.cc:729] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117497b3855a4405b9879b8194a2005d, State: Initialized, Role: FOLLOWER
I20260620 04:37:18.484874 12113 consensus_queue.cc:260] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.486140 12113 ts_tablet_manager.cc:1428] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:37:18.486524 11958 heartbeater.cc:503] Master 127.0.98.62:38857 was elected leader, sending a full tablet report...
W20260620 04:37:18.488345 11959 tablet.cc:2369] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:37:18.593892 12119 raft_consensus.cc:482] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:37:18.594130 12119 raft_consensus.cc:504] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.595506 12119 leader_election.cc:288] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a20f48ea33d6404a838384dec79d5eb2 (127.0.98.3:38927), 20a55204745e4f12bca2662d151dca1d (127.0.98.1:41109)
W20260620 04:37:18.598374 12093 tablet.cc:2369] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:37:18.616273 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.596086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49435 (local address 127.0.98.3:38927)
0620 04:37:18.596247 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.596252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.608489 (+ 12237us) server_negotiation.cc:407] Connection header received
0620 04:37:18.608573 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.608792 (+ 219us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.608863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.608969 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:37:18.609488 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.610661 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.612785 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.613788 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.613895 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.613998 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.616013 (+ 2015us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.616106 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:37:18.616881 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.595743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38927 (local address 127.0.98.2:49435)
0620 04:37:18.603589 (+ 7846us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.603610 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.603972 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.609125 (+ 5153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.609130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.609158 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:37:18.609325 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.609336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.610906 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.610910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.612603 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.612615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.614112 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.614117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.614241 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.614569 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.616505 (+ 1936us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.616593 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7613,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260620 04:37:18.617785 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40aea7abc4b42df95b6141569e3c8ef" candidate_uuid: "117497b3855a4405b9879b8194a2005d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20f48ea33d6404a838384dec79d5eb2" is_pre_election: true
I20260620 04:37:18.618149 12043 raft_consensus.cc:2454] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117497b3855a4405b9879b8194a2005d in term 0.
I20260620 04:37:18.618855 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.596374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49363 (local address 127.0.98.1:41109)
0620 04:37:18.596590 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.596595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.601385 (+ 4790us) server_negotiation.cc:407] Connection header received
0620 04:37:18.601623 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.601627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.601685 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.601740 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:37:18.602193 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.602942 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.615978 (+ 13036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.616719 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.616869 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.616954 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.618632 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.618729 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 04:37:18.618891 11848 leader_election.cc:302] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117497b3855a4405b9879b8194a2005d, a20f48ea33d6404a838384dec79d5eb2; no voters:
I20260620 04:37:18.619208 12119 raft_consensus.cc:2781] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:37:18.619287 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.595743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41109 (local address 127.0.98.2:49363)
0620 04:37:18.601256 (+ 5513us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.601277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:37:18.601481 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:37:18.601910 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:37:18.601913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:37:18.601932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:37:18.602050 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.602057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.603100 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.603103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.615621 (+ 12518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:37:18.615632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.618210 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.618214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:37:18.618318 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.618345 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:37:18.619055 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 04:37:18.619137 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5428,"mutex_wait_us":2143,"thread_start_us":74,"threads_started":1}
I20260620 04:37:18.619302 12119 raft_consensus.cc:482] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:37:18.619365 12119 raft_consensus.cc:3037] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:37:18.623627 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40aea7abc4b42df95b6141569e3c8ef" candidate_uuid: "117497b3855a4405b9879b8194a2005d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a55204745e4f12bca2662d151dca1d" is_pre_election: true
I20260620 04:37:18.623998 11776 raft_consensus.cc:2454] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117497b3855a4405b9879b8194a2005d in term 0.
I20260620 04:37:18.624406 12119 raft_consensus.cc:504] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.624816 12119 leader_election.cc:288] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [CANDIDATE]: Term 1 election: Requested vote from peers a20f48ea33d6404a838384dec79d5eb2 (127.0.98.3:38927), 20a55204745e4f12bca2662d151dca1d (127.0.98.1:41109)
I20260620 04:37:18.625252 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40aea7abc4b42df95b6141569e3c8ef" candidate_uuid: "117497b3855a4405b9879b8194a2005d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20f48ea33d6404a838384dec79d5eb2"
I20260620 04:37:18.625411 12043 raft_consensus.cc:3037] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:37:18.627051 12043 raft_consensus.cc:2454] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117497b3855a4405b9879b8194a2005d in term 1.
I20260620 04:37:18.627449 11848 leader_election.cc:302] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117497b3855a4405b9879b8194a2005d, a20f48ea33d6404a838384dec79d5eb2; no voters:
I20260620 04:37:18.627730 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40aea7abc4b42df95b6141569e3c8ef" candidate_uuid: "117497b3855a4405b9879b8194a2005d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a55204745e4f12bca2662d151dca1d"
I20260620 04:37:18.627878 11776 raft_consensus.cc:3037] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:37:18.629462 11776 raft_consensus.cc:2454] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117497b3855a4405b9879b8194a2005d in term 1.
I20260620 04:37:18.629688 12119 raft_consensus.cc:2781] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:37:18.629920 12119 raft_consensus.cc:686] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 1 LEADER]: Becoming Leader. State: Replica: 117497b3855a4405b9879b8194a2005d, State: Running, Role: LEADER
I20260620 04:37:18.630316 12119 consensus_queue.cc:237] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } }
I20260620 04:37:18.634037 11608 catalog_manager.cc:5495] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d reported cstate change: term changed from 0 to 1, leader changed from <none> to 117497b3855a4405b9879b8194a2005d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "117497b3855a4405b9879b8194a2005d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117497b3855a4405b9879b8194a2005d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40041 } health_report { overall_health: HEALTHY } } }
W20260620 04:37:18.640455 11823 tablet.cc:2369] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:37:18.673897 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.668881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.2:40041)
0620 04:37:18.669009 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:37:18.669014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:37:18.669031 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:37:18.669122 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:37:18.669125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:37:18.669198 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:37:18.669316 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:37:18.669729 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.670476 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.671693 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:37:18.672021 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:37:18.672120 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:37:18.672568 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:37:18.673117 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:37:18.673233 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:37:18.673689 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 04:37:18.673750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:37:18.693706 11776 raft_consensus.cc:1261] T a40aea7abc4b42df95b6141569e3c8ef P 20a55204745e4f12bca2662d151dca1d [term 1 FOLLOWER]: Refusing update from remote peer 117497b3855a4405b9879b8194a2005d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:37:18.694561 12119 consensus_queue.cc:1035] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a55204745e4f12bca2662d151dca1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:37:18.697561 12043 raft_consensus.cc:1261] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 1 FOLLOWER]: Refusing update from remote peer 117497b3855a4405b9879b8194a2005d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:37:18.698309 12119 consensus_queue.cc:1035] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20f48ea33d6404a838384dec79d5eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:37:18.715312 12127 mvcc.cc:205] Tried to move back new op lower bound from 7298786257670316032 to 7298786257432997888. Current Snapshot: MvccSnapshot[applied={T|T < 7298786257670316032}]
I20260620 04:37:18.725661 11607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:37:18.725852 11607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:37:18.728463 11607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f253a2b57d846d08f2c6f13db265044: Sending DeleteTablet for 3 replicas of tablet a40aea7abc4b42df95b6141569e3c8ef
I20260620 04:37:18.729492 12026 tablet_service.cc:1514] Processing DeleteTablet for tablet a40aea7abc4b42df95b6141569e3c8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39118
I20260620 04:37:18.729841 11892 tablet_service.cc:1514] Processing DeleteTablet for tablet a40aea7abc4b42df95b6141569e3c8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260620 04:37:18.730077 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet a40aea7abc4b42df95b6141569e3c8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40954
I20260620 04:37:18.731508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 11689
I20260620 04:37:18.739192 12137 tablet_replica.cc:330] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: stopping tablet replica
I20260620 04:37:18.739483 12136 tablet_replica.cc:330] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: stopping tablet replica
I20260620 04:37:18.739693 12137 raft_consensus.cc:2229] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:37:18.739750 12136 raft_consensus.cc:2229] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:37:18.740074 12136 raft_consensus.cc:2258] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:37:18.740324 12137 raft_consensus.cc:2258] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:37:18.742249 12137 ts_tablet_manager.cc:1905] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:37:18.749071 12136 ts_tablet_manager.cc:1905] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:37:18.752075 12136 ts_tablet_manager.cc:1918] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:37:18.752215 12136 log.cc:1196] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/a40aea7abc4b42df95b6141569e3c8ef
I20260620 04:37:18.752586 12136 ts_tablet_manager.cc:1939] T a40aea7abc4b42df95b6141569e3c8ef P a20f48ea33d6404a838384dec79d5eb2: Deleting consensus metadata
I20260620 04:37:18.754042 11595 catalog_manager.cc:4841] TS a20f48ea33d6404a838384dec79d5eb2 (127.0.98.3:38927): tablet a40aea7abc4b42df95b6141569e3c8ef (table test-table [id=b8632f99dda2492487a74fce9f356abb]) successfully deleted
W20260620 04:37:18.761749 11592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41109 (error 108)
I20260620 04:37:18.764089 12137 ts_tablet_manager.cc:1918] T a40aea7abc4b42df95b6141569e3c8ef P 117497b3855a4405b9879b8194a2005d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:37:18.764446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 11828
I20260620 04:37:18.766124 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.765733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41109 (local address 127.0.0.1:40966)
0620 04:37:18.765969 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.766038 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260620 04:37:18.766340 11592 catalog_manager.cc:4584] TS 20a55204745e4f12bca2662d151dca1d (127.0.98.1:41109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a40aea7abc4b42df95b6141569e3c8ef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41109: connect: Connection refused (error 111)
I20260620 04:37:18.772383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 11963
I20260620 04:37:18.772799 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:37:18.772429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40041 (local address 127.0.0.1:37682)
0620 04:37:18.772628 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 04:37:18.772688 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 04:37:18.780054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 11576
I20260620 04:37:18.786680 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:37:19.132323 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:37:19.132473 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:37:19.478148 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:37:19.478 UTC [11162] LOG: received smart shutdown request
2026-06-20 04:37:19.482 UTC [11162] LOG: background worker "logical replication launcher" (PID 11171) exited with exit code 1
2026-06-20 04:37:19.482 UTC [11166] LOG: shutting down
2026-06-20 04:37:19.671 UTC [11162] LOG: database system is shut down
2026-06-20T04:37:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49392 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:37:19 dist-test-slave-zmf4 krb5kdc[12148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:37:19 dist-test-slave-zmf4 krb5kdc[12148](info): set up 2 sockets
Jun 20 04:37:19 dist-test-slave-zmf4 krb5kdc[12148](info): commencing operation
krb5kdc: starting...
W20260620 04:37:20.864832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:37:20 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:37:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:37:21.040381 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGAOikI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:37:23.777 UTC [12178] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:37:23.777 UTC [12178] LOG: listening on IPv4 address "127.0.98.20", port 40237
2026-06-20 04:37:23.781 UTC [12178] LOG: listening on Unix socket "/tmp/.s.PGSQL.40237"
2026-06-20 04:37:23.816 UTC [12181] LOG: database system was shut down at 2026-06-20 04:37:22 UTC
2026-06-20 04:37:23.820 UTC [12178] LOG: database system is ready to accept connections
I20260620 04:37:24.804111 392 mini_postgres.cc:96] Postgres bound to 40237
2026-06-20 04:37:24.810 UTC [12189] FATAL: database "slave" does not exist
127.0.98.20:40237 - accepting connections
I20260620 04:37:24.811088 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:37:24.826841 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:37:25.362150 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:37:25.362244 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin
2026-06-20 04:37:25,644 [I] DB FLAVOR :POSTGRES
2026-06-20 04:37:25,645 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:37:25,645 [I] Checking connection..
2026-06-20 04:37:25,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:37:25,920 [I] Checking connection passed.
2026-06-20 04:37:25,921 [I] --------- Verifying version history table ---------
2026-06-20 04:37:25,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:37:26,177 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:37:26,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:37:26,429 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:37:26,429 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:37:26,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:37:26.697 UTC [12253] WARNING: there is no transaction in progress
2026-06-20 04:37:26,703 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:37:26,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:37:26,984 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:37:26,984 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:37:26,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:37:27,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:37:27,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zmf4.c.gcp-upstream.internal','N') ;"
2026-06-20 04:37:27,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:37:27,997 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:37:27,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:37:28,237 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:37:28,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:37:28,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:37:28,761 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:37:28,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:37:29.772 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.790 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.809 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.827 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.848 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.870 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.941 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.949 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.966 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.982 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:29.993 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30.007 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30.018 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30.024 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30.031 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30.038 UTC [12361] WARNING: there is no transaction in progress
2026-06-20 04:37:30,760 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:37:30,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:37:31,003 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:37:31,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:37:31,282 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:37:31,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:37:31,530 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:37:31,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:37:31,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:37:32,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zmf4.c.gcp-upstream.internal';"
2026-06-20 04:37:32,257 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:37:32,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:37:32,511 [I] DB_PATCHES have already been applied
I20260620 04:37:32.516762 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGAOikI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:37:32.516830 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:37:32.520277 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42089
Jun 20, 2026 4:37:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:37:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:37:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:37:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:37:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:37:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:37:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:37:33.689499 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.878s user 0.001s sys 0.009s
Jun 20, 2026 4:37:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:37:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 4:37:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:37:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:37:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:37:56.040143 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:37:56 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930276, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:37:56.140712 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:37:56.151623 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:37:57.126377 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:37:57.126478 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms
I20260620 04:37:57.126499 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40237
2026-06-20 04:37:57,311 [I] DB FLAVOR :POSTGRES
2026-06-20 04:37:57,312 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:37:57,312 [I] Checking connection
2026-06-20 04:37:57,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:37:57,556 [I] connection success
2026-06-20 04:37:57,557 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:37:57,557 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:37:57,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:37:57,799 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:37:57,800 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:37:57,800 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:37:57,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:37:58,076 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:37:58.592530 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:37:58.799587 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:37:58.845989 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:37:58 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930276, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:37:59.256997 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:37:59.257154 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGAOikI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGAOikI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:37:59.257185 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:37:59.260854 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12542
I20260620 04:37:59.260928 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53941
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:38:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 04:38:01 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781930281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:01.568737 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.428s user 0.004s sys 0.162s
I20260620 04:38:01.568871 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:38:01.568939 392 mini_ranger_kms.cc:401] 127.0.98.20:53941/kms/v1/keys
Jun 20 04:38:01 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:38:02 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:38:04 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:38:04.528409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40581
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:04.639990 12592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:04.640278 12592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:04.640369 12592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:04.644536 12592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:04.644598 12592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:04.644618 12592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:04.644634 12592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:04.644690 12592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:04.650211 12592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53941/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40581
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:04.651402 12592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:04.652508 12592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:04.661780 12598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:04.661842 12600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:04.661993 12597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:04.661988 12592 server_base.cc:1034] running on GCE node
I20260620 04:38:04.662606 12592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:04.663704 12592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:04.664920 12592 hybrid_clock.cc:648] HybridClock initialized: now 1781930284664902 us; error 37 us; skew 500 ppm
Jun 20 04:38:04 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:04.671984 12592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:04.673050 12592 webserver.cc:460] Webserver started at http://127.0.98.62:44167/ using document root <none> and password file <none>
I20260620 04:38:04.673604 12592 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:04.673682 12592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:04.673899 12592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:38:04 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:04.734596 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856"
format_stamp: "Formatted at 2026-06-20 04:38:04 on dist-test-slave-zmf4"
server_key: "ec47e1894bbfc241a32fa2f341b5a59d"
server_key_iv: "ce9ca02362d87ec69b14730308100845"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:04.735348 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856"
format_stamp: "Formatted at 2026-06-20 04:38:04 on dist-test-slave-zmf4"
server_key: "ec47e1894bbfc241a32fa2f341b5a59d"
server_key_iv: "ce9ca02362d87ec69b14730308100845"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:04.740330 12592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 04:38:04.758950 12607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:04.761018 12592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:04.761173 12592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856"
format_stamp: "Formatted at 2026-06-20 04:38:04 on dist-test-slave-zmf4"
server_key: "ec47e1894bbfc241a32fa2f341b5a59d"
server_key_iv: "ce9ca02362d87ec69b14730308100845"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:04.761325 12592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:04.783396 12592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:04.786846 12592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:04.787314 12592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:04.795835 12592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40581
I20260620 04:38:04.795840 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40581 every 8 connection(s)
I20260620 04:38:04.797044 12592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:04.799851 12660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:04.805578 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Bootstrap starting.
I20260620 04:38:04.806461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 12592
I20260620 04:38:04.806597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:38:04.806931 392 external_mini_cluster.cc:1428] Setting key c66dcba36195e86b890588d96b9f8fb7
I20260620 04:38:04.808931 12660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:04.809675 12660 log.cc:824] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:04.811758 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: No bootstrap required, opened a new log
I20260620 04:38:04.816412 12660 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:04.816727 12660 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:04.816849 12660 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a9a323c2c84f3cb2a0fe63c40f7856, State: Initialized, Role: FOLLOWER
I20260620 04:38:04.817401 12660 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:04.817572 12660 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:04.817673 12660 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 04:38:04 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:04.817787 12660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:04.818893 12660 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:04.819247 12660 leader_election.cc:302] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a9a323c2c84f3cb2a0fe63c40f7856; no voters:
I20260620 04:38:04.819628 12660 leader_election.cc:288] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:38:04.820348 12665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:04.820588 12665 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 LEADER]: Becoming Leader. State: Replica: d0a9a323c2c84f3cb2a0fe63c40f7856, State: Running, Role: LEADER
I20260620 04:38:04.821025 12660 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:04.821727 12665 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:04.823733 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a9a323c2c84f3cb2a0fe63c40f7856. Latest consensus state: current_term: 1 leader_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } } }
I20260620 04:38:04.823889 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:04.823899 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:04.808092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.0.98.62:40581)
0620 04:38:04.808583 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:04.808601 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:04.808644 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:38:04.809502 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:04.809526 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:04.809946 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:04.810322 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:04.811201 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:04.812955 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:04.814386 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:04.815120 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:04.815270 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:04.818462 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:04.818469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:04.818486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:04.818518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:04.820591 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:04.821095 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:04.821101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:04.821107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:04.821239 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:04.821639 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:04.821643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:04.821645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:04.822188 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:04.822379 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:04.822672 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 04:38:04.822951 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":190,"threads_started":1}
I20260620 04:38:04.824090 12660 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:38:04.825493 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } } }
I20260620 04:38:04.825603 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:04.829186 12679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:04.832890 12679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:04.842048 12679 catalog_manager.cc:1347] Generated new cluster ID: d6a07936fc534742a63da2050a89f880
I20260620 04:38:04.842128 12679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:04.863658 12679 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:38:04.865100 12679 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:04.873553 12679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Generated new TSK 0
I20260620 04:38:04.874542 12679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:38:05.643643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:05.809262 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:05.809537 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:05.809643 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:05.814903 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:05.815035 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:05.815212 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:05.823313 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:05.824735 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:05.826094 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:05.842344 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:05.843307 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:05.843547 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:05.843825 12705 server_base.cc:1034] running on GCE node
I20260620 04:38:05.844357 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:05.845119 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:05.852483 12705 hybrid_clock.cc:648] HybridClock initialized: now 1781930285852405 us; error 83 us; skew 500 ppm
Jun 20 04:38:05 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:05.856397 12705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:05.857553 12705 webserver.cc:460] Webserver started at http://127.0.98.1:39707/ using document root <none> and password file <none>
I20260620 04:38:05.858299 12705 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:05.858457 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:05.858728 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:38:05 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:05.874826 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "54e507948f904727844ec86bfbaac733"
format_stamp: "Formatted at 2026-06-20 04:38:05 on dist-test-slave-zmf4"
server_key: "3c0e4956c395c52680b5c8be60dfd445"
server_key_iv: "aacd87034b4eeaa3c772cabf4306b442"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:05.875450 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54e507948f904727844ec86bfbaac733"
format_stamp: "Formatted at 2026-06-20 04:38:05 on dist-test-slave-zmf4"
server_key: "3c0e4956c395c52680b5c8be60dfd445"
server_key_iv: "aacd87034b4eeaa3c772cabf4306b442"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:05.885681 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260620 04:38:05.920640 12722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:05.922643 12705 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.001s sys 0.000s
I20260620 04:38:05.922782 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "54e507948f904727844ec86bfbaac733"
format_stamp: "Formatted at 2026-06-20 04:38:05 on dist-test-slave-zmf4"
server_key: "3c0e4956c395c52680b5c8be60dfd445"
server_key_iv: "aacd87034b4eeaa3c772cabf4306b442"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:05.922922 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:05.947384 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:05.951117 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:05.951375 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:05.952158 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:05.953298 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:05.953404 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:05.953539 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:05.953629 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:05.964206 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43785
I20260620 04:38:05.965412 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:05.970940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 12705
I20260620 04:38:05.971200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:38:05.971692 392 external_mini_cluster.cc:1428] Setting key 1624637ce9bfef0caa9fe2944af5fe6f
I20260620 04:38:05.974071 12837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43785 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 04:38:05 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:05.996097 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:05.976440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.1:42729)
0620 04:38:05.977804 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:05.977866 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:05.978909 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:05.979751 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:05.979760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:05.980106 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:05.980861 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:05.980883 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:05.982564 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:05.982568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:05.984097 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:05.984111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:05.984843 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:05.984850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:05.985091 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:05.985776 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:05.985801 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:05.989405 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:05.991877 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:05.991887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:05.991909 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:05.992454 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:05.993116 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:05.993120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:05.993122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:05.993341 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:05.993911 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:05.993921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:05.994321 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 04:38:05.994543 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 04:38:05.994937 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":137,"threads_started":1}
I20260620 04:38:05.997139 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:05.977604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42729 (local address 127.0.98.62:40581)
0620 04:38:05.979348 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:05.979355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:05.979369 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:38:05.979409 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:05.979412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:05.979458 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:05.981030 (+ 1572us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:05.981173 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:05.982321 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:05.984287 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:05.984658 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:05.985429 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:05.989635 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:05.989639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:05.989643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:05.989678 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:05.991734 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:05.992752 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:05.992758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:05.992763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:05.992985 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:05.993470 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:05.993476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:05.993481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:05.993760 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:05.996716 (+ 2956us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:05.996826 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:38:05.996971 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":95,"threads_started":1}
I20260620 04:38:05.998296 12838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:05.998543 12838 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:05.999240 12838 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:06.000943 12624 ts_manager.cc:194] Registered new tserver with Master: 54e507948f904727844ec86bfbaac733 (127.0.98.1:43785)
I20260620 04:38:06.002568 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:38:06.007563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:38:06.164747 12844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:06.165292 12844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:06.165409 12844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:06.170025 12844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:06.170114 12844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:06.170224 12844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:06.177404 12844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:06.178629 12844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:06.179877 12844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:06.190709 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.190765 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.190855 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:06.191725 12844 server_base.cc:1034] running on GCE node
I20260620 04:38:06.192370 12844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:06.193164 12844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:06.194410 12844 hybrid_clock.cc:648] HybridClock initialized: now 1781930286194373 us; error 56 us; skew 500 ppm
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:06.197482 12844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:06.198519 12844 webserver.cc:460] Webserver started at http://127.0.98.2:34969/ using document root <none> and password file <none>
I20260620 04:38:06.199343 12844 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:06.199452 12844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:06.199712 12844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:06.215684 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "1d80a4f92088429ba5880ad6dc299f5b"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "c5712946aecb1800e72d907a73b17cd7"
server_key_iv: "14649f40d74cf37d0c6f5708acee6bfa"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.220388 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1d80a4f92088429ba5880ad6dc299f5b"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "c5712946aecb1800e72d907a73b17cd7"
server_key_iv: "14649f40d74cf37d0c6f5708acee6bfa"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.225571 12844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 04:38:06.243022 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.244894 12844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:06.245024 12844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "1d80a4f92088429ba5880ad6dc299f5b"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "c5712946aecb1800e72d907a73b17cd7"
server_key_iv: "14649f40d74cf37d0c6f5708acee6bfa"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.245158 12844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:06.268134 12844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:06.272564 12844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:06.272779 12844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:06.273531 12844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:06.274557 12844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:06.274669 12844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.274765 12844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:06.274827 12844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.285764 12844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41897
I20260620 04:38:06.285856 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41897 every 8 connection(s)
I20260620 04:38:06.287016 12844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:06.289366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 12844
I20260620 04:38:06.289502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:38:06.289783 392 external_mini_cluster.cc:1428] Setting key ef5b036c84e1322acd07ba50599b56fd
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:38:06.303476 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:06.289019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45655 (local address 127.0.98.62:40581)
0620 04:38:06.289159 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:06.289166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:06.290620 (+ 1454us) server_negotiation.cc:407] Connection header received
0620 04:38:06.292930 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:06.292934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:06.292985 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:06.293092 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:06.294268 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.295161 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.296776 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.297010 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.297123 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:06.299490 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:06.299493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:06.299495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:06.299520 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:06.301075 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:06.301705 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:06.301710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:06.301712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:06.301799 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:06.302150 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:06.302153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:06.302154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:06.302450 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:06.302557 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:06.303143 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 04:38:06.303272 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:38:06.304544 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:06.289900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.2:45655)
0620 04:38:06.290458 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:06.290517 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:06.292690 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:06.293267 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:06.293275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:06.293541 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:06.294101 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:06.294122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.295327 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.295333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:06.296640 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:06.296649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.297163 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.297167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:06.297349 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:06.297966 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:06.297983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:06.299306 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:06.301196 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:06.301203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:06.301221 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:06.301604 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:06.301899 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:06.301902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:06.301904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:06.302049 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:06.302567 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:06.302575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:06.302888 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 04:38:06.303104 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 04:38:06.303436 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260620 04:38:06.307056 12974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:06.307339 12974 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:06.307991 12974 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:06.309370 12624 ts_manager.cc:194] Registered new tserver with Master: 1d80a4f92088429ba5880ad6dc299f5b (127.0.98.2:41897)
I20260620 04:38:06.309980 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:38:06.332566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 04:38:06.499607 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:06.500368 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:06.500494 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:06.506289 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:06.506408 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:06.506521 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:06.514113 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:06.516592 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:06.518771 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:06.535115 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.536465 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.536790 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:06.537551 12979 server_base.cc:1034] running on GCE node
I20260620 04:38:06.538038 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:06.538784 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:06.540016 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781930286539979 us; error 53 us; skew 500 ppm
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:06.544085 12979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:06.545720 12979 webserver.cc:460] Webserver started at http://127.0.98.3:45793/ using document root <none> and password file <none>
I20260620 04:38:06.546517 12979 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:06.546666 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:06.546952 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:06.566586 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "79429f37ac744ae88404babbe6468fbd"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "37c172cf74e53b2fe74c9f6303b69fe3"
server_key_iv: "19feb22debef6092128e5d23a2751462"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.567402 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79429f37ac744ae88404babbe6468fbd"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "37c172cf74e53b2fe74c9f6303b69fe3"
server_key_iv: "19feb22debef6092128e5d23a2751462"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.572463 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 04:38:06.597528 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.599339 12979 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260620 04:38:06.599525 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "79429f37ac744ae88404babbe6468fbd"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "37c172cf74e53b2fe74c9f6303b69fe3"
server_key_iv: "19feb22debef6092128e5d23a2751462"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.599681 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:06.627315 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:06.631095 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:06.631299 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:06.632086 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:06.633139 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:06.633203 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.633255 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:06.633294 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.644105 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39427
I20260620 04:38:06.645207 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39427 every 8 connection(s)
I20260620 04:38:06.646345 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:06.654680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 12979
I20260620 04:38:06.654816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:38:06.655100 392 external_mini_cluster.cc:1428] Setting key 1deb58e55ecf1105cd66b549299cb5c9
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:06.663868 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:06.650236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41973 (local address 127.0.98.62:40581)
0620 04:38:06.650366 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:06.650370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:06.650388 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:38:06.651145 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:06.651150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:06.651205 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:06.651298 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:06.652752 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.653692 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.655935 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.656321 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.656412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:06.659562 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:06.659566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:06.659569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:06.659597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:06.661371 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:06.662006 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:06.662009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:06.662012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:06.662104 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:06.662537 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:06.662541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:06.662543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:06.662767 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:06.662875 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:06.663560 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 04:38:06.663689 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:38:06.664891 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:06.649076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.3:41973)
0620 04:38:06.649696 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:06.649752 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:06.650890 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:06.651530 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:06.651539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:06.651886 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:06.652582 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:06.652603 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.653845 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.653850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:06.655270 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:06.655281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:06.656618 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:06.656622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:06.656831 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:06.657557 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:06.657578 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:06.659364 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:06.661501 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:06.661507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:06.661525 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:06.661886 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:06.662212 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:06.662215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:06.662217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:06.662423 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:06.662927 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:06.662934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:06.663279 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 04:38:06.663492 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 04:38:06.663824 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":187,"threads_started":1}
I20260620 04:38:06.666333 13108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:06.666584 13108 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:06.667199 13108 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:06.668375 12624 ts_manager.cc:194] Registered new tserver with Master: 79429f37ac744ae88404babbe6468fbd (127.0.98.3:39427)
I20260620 04:38:06.669112 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41973
I20260620 04:38:06.671368 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:06.672312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 12705
I20260620 04:38:06.679870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 12844
I20260620 04:38:06.687518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 12979
I20260620 04:38:06.694674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 12592
I20260620 04:38:06.702208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40581
--webserver_interface=127.0.98.62
--webserver_port=44167
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40581
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:06.814759 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:06.815011 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:06.815050 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:06.819231 13111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:06.819288 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:06.819307 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:06.819324 13111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:06.819339 13111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:06.824738 13111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40581
--ranger_config_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:06.825733 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:06.826727 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:06.835070 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.835078 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:06.835078 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:06.835466 13111 server_base.cc:1034] running on GCE node
I20260620 04:38:06.836005 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:06.837137 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:06.838330 13111 hybrid_clock.cc:648] HybridClock initialized: now 1781930286838296 us; error 42 us; skew 500 ppm
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:06.841077 13111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:06.841903 13111 webserver.cc:460] Webserver started at http://127.0.98.62:44167/ using document root <none> and password file <none>
I20260620 04:38:06.842378 13111 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:06.842438 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:38:06 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:06.922914 13111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:06.925283 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:06.926602 13111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:06.926717 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856"
format_stamp: "Formatted at 2026-06-20 04:38:04 on dist-test-slave-zmf4"
server_key: "ec47e1894bbfc241a32fa2f341b5a59d"
server_key_iv: "ce9ca02362d87ec69b14730308100845"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:06.927318 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:06.943128 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:06.946257 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:06.946467 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:06.954319 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40581
I20260620 04:38:06.954382 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40581 every 8 connection(s)
I20260620 04:38:06.955375 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:06.959614 13180 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:38:06.960345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13111
I20260620 04:38:06.960842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43785
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40581
--webserver_port=39707
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:06.961755 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Bootstrap starting.
I20260620 04:38:06.978433 13180 log.cc:824] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:06.984939 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:06.985348 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Bootstrap complete.
I20260620 04:38:06.989513 13180 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:06.989885 13180 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a9a323c2c84f3cb2a0fe63c40f7856, State: Initialized, Role: FOLLOWER
I20260620 04:38:06.990402 13180 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:06.990513 13180 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:06.990595 13180 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:06.990691 13180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:38:06.992635 13180 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:06.992970 13180 leader_election.cc:302] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a9a323c2c84f3cb2a0fe63c40f7856; no voters:
I20260620 04:38:06.993290 13180 leader_election.cc:288] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:38:06.993374 13184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:38:06.993587 13184 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [term 2 LEADER]: Becoming Leader. State: Replica: d0a9a323c2c84f3cb2a0fe63c40f7856, State: Running, Role: LEADER
I20260620 04:38:06.993875 13184 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } }
I20260620 04:38:06.994504 13180 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:06.994911 13184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a9a323c2c84f3cb2a0fe63c40f7856. Latest consensus state: current_term: 2 leader_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } } }
I20260620 04:38:06.995019 13184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:06.995352 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a323c2c84f3cb2a0fe63c40f7856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40581 } } }
I20260620 04:38:06.995528 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:06.995388 13188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:06.998966 13188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:06.999503 13180 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:38:06.999744 13188 catalog_manager.cc:1259] Loaded cluster ID: d6a07936fc534742a63da2050a89f880
I20260620 04:38:06.999962 13188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:07.005107 13188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:07.005975 13188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Loaded TSK: 0
I20260620 04:38:07.007270 13188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:38:07.102813 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:07.103147 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:07.103211 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:07.109563 13182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:07.109642 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:07.109732 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:07.117830 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39707
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:07.119177 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:07.120556 13182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:07.133544 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.133896 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.133952 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:07.134155 13182 server_base.cc:1034] running on GCE node
I20260620 04:38:07.134984 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:07.135815 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:07.139174 13182 hybrid_clock.cc:648] HybridClock initialized: now 1781930287139029 us; error 154 us; skew 500 ppm
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:07.143054 13182 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:07.144078 13182 webserver.cc:460] Webserver started at http://127.0.98.1:39707/ using document root <none> and password file <none>
I20260620 04:38:07.144744 13182 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:07.144837 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:07.169430 13182 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:38:07.172505 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.174292 13182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:38:07.174434 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "54e507948f904727844ec86bfbaac733"
format_stamp: "Formatted at 2026-06-20 04:38:05 on dist-test-slave-zmf4"
server_key: "3c0e4956c395c52680b5c8be60dfd445"
server_key_iv: "aacd87034b4eeaa3c772cabf4306b442"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:07.175202 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:07.192641 13182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:07.197084 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:07.197293 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:07.198024 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:07.199103 13182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:07.199162 13182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.199215 13182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:07.199239 13182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.212459 13182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43785
I20260620 04:38:07.213763 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:07.217731 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43785 every 8 connection(s)
I20260620 04:38:07.223743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13182
I20260620 04:38:07.224256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40581
--webserver_port=34969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:07.260434 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.224786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59351 (local address 127.0.98.62:40581)
0620 04:38:07.228521 (+ 3735us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:07.228548 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:07.231622 (+ 3074us) server_negotiation.cc:407] Connection header received
0620 04:38:07.236366 (+ 4744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:07.236383 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:07.236912 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:07.237303 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:07.239116 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.240695 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.242590 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.243417 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.243616 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.247134 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:07.247141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:07.247161 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:07.247202 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:07.256133 (+ 8931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.256938 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.256945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.256950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.257118 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.257542 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.257546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.257547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.258232 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:07.258367 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:07.259121 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 04:38:07.259418 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3518,"thread_start_us":139,"threads_started":1}
I20260620 04:38:07.260445 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.222236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.1:59351)
0620 04:38:07.230797 (+ 8561us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:07.230873 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:07.235595 (+ 4722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:07.237843 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:07.237852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:07.238231 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:07.238902 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.238923 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.241016 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.241030 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.242444 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.242457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.243885 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.243892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.244121 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.244914 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:07.244941 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:07.246864 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:07.256344 (+ 9480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.256351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.256369 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.256781 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.257222 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.257225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.257227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.257418 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.258403 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:07.258410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:07.258752 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 04:38:07.258998 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 04:38:07.259379 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8292,"thread_start_us":131,"threads_started":1}
I20260620 04:38:07.264364 13342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:07.264616 13342 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.265192 13342 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.267494 13342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:38:07.267611 13342 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.267781 13342 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:07.268903 13342 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.269063 13342 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.269866 13342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:38:07.382763 13346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:07.383097 13346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:07.383167 13346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:07.388972 13346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:07.389050 13346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:07.389140 13346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:07.396051 13346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34969
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:07.397246 13346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:07.398264 13346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:38:07.410856 13346 server_base.cc:1034] running on GCE node
W20260620 04:38:07.411201 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.411311 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.412359 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:07.412849 13346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:07.413832 13346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:07.418905 13346 hybrid_clock.cc:648] HybridClock initialized: now 1781930287418885 us; error 33 us; skew 500 ppm
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:07.423199 13346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:07.424386 13346 webserver.cc:460] Webserver started at http://127.0.98.2:34969/ using document root <none> and password file <none>
I20260620 04:38:07.425119 13346 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:07.425230 13346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:07.456597 13346 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.001s
I20260620 04:38:07.459416 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.460837 13346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:07.460991 13346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "1d80a4f92088429ba5880ad6dc299f5b"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "c5712946aecb1800e72d907a73b17cd7"
server_key_iv: "14649f40d74cf37d0c6f5708acee6bfa"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:07.461576 13346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:07.484978 13346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:07.488984 13346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:07.489216 13346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:07.489972 13346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:07.491119 13346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:07.491190 13346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.491252 13346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:07.491292 13346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.503435 13346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41897
I20260620 04:38:07.503643 13475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41897 every 8 connection(s)
I20260620 04:38:07.505187 13346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:07.508992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13346
I20260620 04:38:07.509505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39427
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40581
--webserver_port=45793
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:07.526003 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.511077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32883 (local address 127.0.98.62:40581)
0620 04:38:07.511247 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:07.511252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:07.511884 (+ 632us) server_negotiation.cc:407] Connection header received
0620 04:38:07.513119 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:07.513123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:07.513188 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:07.513247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:07.515202 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.516042 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.517861 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.518194 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.518277 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.521448 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:07.521452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:07.521455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:07.521485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:07.523241 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.523961 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.523965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.523967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.524052 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.524519 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.524524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.524527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.524775 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:07.524861 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:07.525618 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 04:38:07.525761 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:38:07.528085 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.507686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.2:32883)
0620 04:38:07.511687 (+ 4001us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:07.511752 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:07.512862 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:07.513878 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:07.513888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:07.514345 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:07.515003 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.515021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.516292 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.516295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.517716 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.517727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.518493 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.518496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.518693 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.519399 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:07.519420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:07.521195 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:07.523392 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.523398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.523495 (+ 97us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.523840 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.524156 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.524158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.524160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.524400 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.524933 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:07.524941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:07.525310 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 04:38:07.526569 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0620 04:38:07.526926 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3767,"thread_start_us":136,"threads_started":1}
I20260620 04:38:07.529769 13476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:07.530022 13476 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.530663 13476 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.532688 13476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:38:07.532827 13476 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.533028 13476 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:07.534032 13476 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.534227 13476 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.535121 13476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:38:07.684532 13479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:07.684907 13479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:07.684968 13479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:07.690896 13479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:07.690963 13479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:07.691048 13479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:07.698747 13479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45793
--tserver_master_addrs=127.0.98.62:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:07.700035 13479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:07.701288 13479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:07.714478 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.714471 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:07.714639 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:07.715029 13479 server_base.cc:1034] running on GCE node
I20260620 04:38:07.715566 13479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:07.716300 13479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:07.726373 13479 hybrid_clock.cc:648] HybridClock initialized: now 1781930287726294 us; error 98 us; skew 500 ppm
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:07.730087 13479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:07.731137 13479 webserver.cc:460] Webserver started at http://127.0.98.3:45793/ using document root <none> and password file <none>
I20260620 04:38:07.731809 13479 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:07.731912 13479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:07.757581 13479 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 04:38:07.760699 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.762332 13479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:07.762458 13479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "79429f37ac744ae88404babbe6468fbd"
format_stamp: "Formatted at 2026-06-20 04:38:06 on dist-test-slave-zmf4"
server_key: "37c172cf74e53b2fe74c9f6303b69fe3"
server_key_iv: "19feb22debef6092128e5d23a2751462"
server_key_version: "kuduclusterkey@0"
I20260620 04:38:07.763182 13479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:07.791180 13479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:07.795236 13479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:07.795501 13479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:07.796245 13479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:07.797314 13479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:07.797363 13479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.797415 13479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:07.797436 13479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:07.810513 13479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39427
I20260620 04:38:07.812011 13479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:07.812945 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39427 every 8 connection(s)
I20260620 04:38:07.820887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13479
Jun 20 04:38:07 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:07.830905 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.815779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60881 (local address 127.0.98.62:40581)
0620 04:38:07.815910 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:07.815916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:07.817590 (+ 1674us) server_negotiation.cc:407] Connection header received
0620 04:38:07.818767 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:07.818771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:07.818828 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:07.818878 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:07.820461 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.821251 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.823034 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.823390 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.823506 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.826547 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:07.826551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:07.826554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:07.826581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:07.828386 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.829065 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.829069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.829071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.829155 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.829558 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.829561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.829563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.829771 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:07.829876 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:07.830599 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 04:38:07.830719 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:38:07.832762 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.816288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40581 (local address 127.0.98.3:60881)
0620 04:38:07.817432 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:07.817486 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:07.818527 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:07.819285 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:07.819294 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:07.819629 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:07.820300 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.820318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.821407 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.821411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.822886 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:07.822897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.823676 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.823680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:07.823889 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.824601 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:07.824622 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:07.826345 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:07.828541 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.828547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.828564 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.828944 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.829260 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:07.829263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:07.829265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:07.829446 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:07.829945 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:07.829951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:07.830290 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 04:38:07.831319 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0620 04:38:07.831676 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":146,"threads_started":1}
I20260620 04:38:07.833532 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:07.821714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.98.62:40581)
0620 04:38:07.823469 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:07.823473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:07.823485 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:38:07.823519 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:07.823522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:07.823564 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:07.823638 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:07.824641 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.826118 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.827246 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:07.827568 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:07.827656 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:07.829341 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:07.829345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:07.829348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:07.829371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:07.831023 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.831487 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.831491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.831492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.831569 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:07.831844 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:07.831847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:07.831849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:07.832034 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:07.832863 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:07.832936 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:38:07.833046 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1683,"thread_start_us":107,"threads_started":1}
I20260620 04:38:07.834587 13610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40581
I20260620 04:38:07.834827 13610 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.835357 13610 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.837989 13610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:38:07.838119 13610 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.838294 13610 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:07.839176 13610 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:07.839350 13610 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
W20260620 04:38:07.840153 13610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 04:38:08 dist-test-slave-zmf4 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781930288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:08.270097 13342 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:08.270407 13342 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:08.271713 13144 ts_manager.cc:194] Registered new tserver with Master: 54e507948f904727844ec86bfbaac733 (127.0.98.1:43785)
I20260620 04:38:08.273429 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59351
I20260620 04:38:08.535354 13476 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:08.535663 13476 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:08.536446 13144 ts_manager.cc:194] Registered new tserver with Master: 1d80a4f92088429ba5880ad6dc299f5b (127.0.98.2:41897)
I20260620 04:38:08.537228 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32883
Jun 20 04:38:08 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:38:08 dist-test-slave-zmf4 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:38:08.840444 13610 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:08.840776 13610 heartbeater.cc:511] Master 127.0.98.62:40581 requested a full tablet report, sending...
I20260620 04:38:08.841720 13144 ts_manager.cc:194] Registered new tserver with Master: 79429f37ac744ae88404babbe6468fbd (127.0.98.3:39427)
I20260620 04:38:08.842685 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60881
I20260620 04:38:08.843415 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:08.857118 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.845599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.0.98.62:40581)
0620 04:38:08.846223 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:08.846228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:08.846580 (+ 352us) server_negotiation.cc:407] Connection header received
0620 04:38:08.846626 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:08.846629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:08.846676 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:08.846758 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:08.847737 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.848636 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.849898 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.850284 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.850397 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.852448 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:08.852452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:08.852456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:08.852484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:08.854532 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:08.855092 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:08.855096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:08.855098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:08.855210 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:08.855567 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:08.855571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:08.855573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:08.855813 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:08.855967 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:08.856771 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 04:38:08.856918 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":108,"threads_started":1}
I20260620 04:38:08.862196 13143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:38:08.868579 13143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:38:08.891213 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.882833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43785 (local address 127.0.0.1:40816)
0620 04:38:08.884754 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:08.884806 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:08.884956 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:08.885787 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:08.885793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:08.885816 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:08.885966 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.885973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.887560 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.887563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.889150 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.889162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.890616 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.890621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.890737 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.890780 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 04:38:08.890914 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 04:38:08.891042 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":100,"threads_started":1}
I20260620 04:38:08.891875 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.882945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.98.1:43785)
0620 04:38:08.883646 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:08.883654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:08.885097 (+ 1443us) server_negotiation.cc:407] Connection header received
0620 04:38:08.885180 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:08.885186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:08.885405 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:08.885583 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:08.886112 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.887414 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.889305 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.890251 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.890385 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.890547 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:08.891598 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0620 04:38:08.891711 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":86,"threads_started":1}
I20260620 04:38:08.895467 13259 tablet_service.cc:1467] Processing CreateTablet for tablet e4092a06d0364d3c9f13a46310acfbff (DEFAULT_TABLE table=test-table [id=3d43bc8e97524c96babd1ea53380e944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:08.897001 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4092a06d0364d3c9f13a46310acfbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:08.905054 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.885605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41897 (local address 127.0.0.1:43984)
0620 04:38:08.888396 (+ 2791us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:08.888409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:08.888492 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:08.900277 (+ 11785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:08.900280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:08.900304 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:08.900416 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.900422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.901947 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.901951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.903219 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.903231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.904532 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.904536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.904657 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.904689 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 04:38:08.904759 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:38:08.904858 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2735,"thread_start_us":94,"threads_started":1}
I20260620 04:38:08.905923 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.885754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.0.98.2:41897)
0620 04:38:08.899496 (+ 13742us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:08.899504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:08.899534 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:38:08.899635 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:08.899641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:08.899898 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:08.900103 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:08.900615 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.901794 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.903383 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.904289 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.905431 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.905605 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:08.905665 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 04:38:08.905770 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11202,"thread_start_us":105,"threads_started":1}
I20260620 04:38:08.912245 13400 tablet_service.cc:1467] Processing CreateTablet for tablet e4092a06d0364d3c9f13a46310acfbff (DEFAULT_TABLE table=test-table [id=3d43bc8e97524c96babd1ea53380e944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:08.913569 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.884240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39427 (local address 127.0.0.1:51116)
0620 04:38:08.886797 (+ 2557us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:08.886811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:08.886899 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:08.888062 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:08.888065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:08.888081 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:08.888163 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.888167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.890311 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.890315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.891405 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:08.891415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.913052 (+ 21637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.913057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:08.913179 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.913209 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 04:38:08.913301 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:38:08.913393 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2503,"thread_start_us":128,"threads_started":1}
I20260620 04:38:08.913663 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4092a06d0364d3c9f13a46310acfbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:08.916700 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:08.884481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.98.3:39427)
0620 04:38:08.884828 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:08.884835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:08.887117 (+ 2282us) server_negotiation.cc:407] Connection header received
0620 04:38:08.887553 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:08.887559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:08.887744 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:08.887882 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:08.888604 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.890184 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.891630 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:08.896566 (+ 4936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:08.896688 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:08.896853 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:08.916336 (+ 19483us) server_negotiation.cc:299] Negotiation successful
0620 04:38:08.916526 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":62,"threads_started":1}
I20260620 04:38:08.918812 13646 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Bootstrap starting.
I20260620 04:38:08.921162 13646 tablet_bootstrap.cc:654] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:08.921032 13544 tablet_service.cc:1467] Processing CreateTablet for tablet e4092a06d0364d3c9f13a46310acfbff (DEFAULT_TABLE table=test-table [id=3d43bc8e97524c96babd1ea53380e944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:08.922001 13646 log.cc:824] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:08.922398 13544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4092a06d0364d3c9f13a46310acfbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:08.930974 13649 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Bootstrap starting.
I20260620 04:38:08.932106 13648 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Bootstrap starting.
I20260620 04:38:08.933606 13649 tablet_bootstrap.cc:654] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:08.934476 13649 log.cc:824] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:08.935411 13646 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: No bootstrap required, opened a new log
I20260620 04:38:08.935707 13646 ts_tablet_manager.cc:1397] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260620 04:38:08.936486 13649 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: No bootstrap required, opened a new log
I20260620 04:38:08.936666 13649 ts_tablet_manager.cc:1397] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:38:08.940750 13649 raft_consensus.cc:348] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.941048 13649 raft_consensus.cc:374] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:08.941102 13649 raft_consensus.cc:729] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79429f37ac744ae88404babbe6468fbd, State: Initialized, Role: FOLLOWER
I20260620 04:38:08.942624 13649 consensus_queue.cc:260] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.940800 13646 raft_consensus.cc:348] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.943040 13646 raft_consensus.cc:374] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:08.943154 13646 raft_consensus.cc:729] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e507948f904727844ec86bfbaac733, State: Initialized, Role: FOLLOWER
I20260620 04:38:08.943728 13646 consensus_queue.cc:260] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.943854 13649 ts_tablet_manager.cc:1428] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:38:08.944941 13646 ts_tablet_manager.cc:1428] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260620 04:38:08.945281 13648 tablet_bootstrap.cc:654] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:08.946503 13342 heartbeater.cc:503] Master 127.0.98.62:40581 was elected leader, sending a full tablet report...
I20260620 04:38:08.952621 13610 heartbeater.cc:503] Master 127.0.98.62:40581 was elected leader, sending a full tablet report...
I20260620 04:38:08.966362 13648 log.cc:824] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:08.970013 13648 tablet_bootstrap.cc:492] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: No bootstrap required, opened a new log
I20260620 04:38:08.970309 13648 ts_tablet_manager.cc:1397] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
W20260620 04:38:08.971249 13343 tablet.cc:2369] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:08.975934 13648 raft_consensus.cc:348] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.976591 13648 raft_consensus.cc:374] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:08.976708 13648 raft_consensus.cc:729] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d80a4f92088429ba5880ad6dc299f5b, State: Initialized, Role: FOLLOWER
I20260620 04:38:08.977277 13648 consensus_queue.cc:260] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:08.978350 13476 heartbeater.cc:503] Master 127.0.98.62:40581 was elected leader, sending a full tablet report...
I20260620 04:38:08.982722 13648 ts_tablet_manager.cc:1428] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260620 04:38:09.013856 13477 tablet.cc:2369] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:38:09.065416 13611 tablet.cc:2369] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:09.185236 13654 raft_consensus.cc:482] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:09.185631 13654 raft_consensus.cc:504] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:09.186903 13654 leader_election.cc:288] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e507948f904727844ec86bfbaac733 (127.0.98.1:43785), 79429f37ac744ae88404babbe6468fbd (127.0.98.3:39427)
I20260620 04:38:09.191773 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.187412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43113 (local address 127.0.98.3:39427)
0620 04:38:09.187555 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:09.187560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:09.187799 (+ 239us) server_negotiation.cc:407] Connection header received
0620 04:38:09.187967 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:09.187974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:09.188034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:09.188133 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:09.188768 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.189592 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.190687 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.191107 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.191196 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:09.191245 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:09.191556 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 04:38:09.191634 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:38:09.191805 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.187230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43785 (local address 127.0.98.2:45077)
0620 04:38:09.187626 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:09.187639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:09.187789 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:09.188139 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:09.188143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:09.188159 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:09.188368 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:09.188375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.189398 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.189402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:09.190341 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:09.190350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.191343 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.191349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:09.191461 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:09.191479 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:38:09.191574 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:38:09.191645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":19,"thread_start_us":134,"threads_started":1}
I20260620 04:38:09.191812 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.187316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45077 (local address 127.0.98.1:43785)
0620 04:38:09.187467 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:09.187472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:09.187735 (+ 263us) server_negotiation.cc:407] Connection header received
0620 04:38:09.187920 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:09.187924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:09.188008 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:09.188111 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:09.188548 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.189304 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.190481 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.191200 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.191319 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:09.191408 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:09.191609 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 04:38:09.191684 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:38:09.191829 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.187240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39427 (local address 127.0.98.2:43113)
0620 04:38:09.187699 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:09.187711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:09.187847 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:09.188505 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:09.188508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:09.188525 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:09.188645 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:09.188651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.189713 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.189715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:09.190539 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:09.190549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.191238 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.191245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:09.191374 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:09.191440 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 04:38:09.191517 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:38:09.191622 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":31,"thread_start_us":146,"threads_started":1}
I20260620 04:38:09.192498 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4092a06d0364d3c9f13a46310acfbff" candidate_uuid: "1d80a4f92088429ba5880ad6dc299f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e507948f904727844ec86bfbaac733" is_pre_election: true
I20260620 04:38:09.192632 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4092a06d0364d3c9f13a46310acfbff" candidate_uuid: "1d80a4f92088429ba5880ad6dc299f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79429f37ac744ae88404babbe6468fbd" is_pre_election: true
I20260620 04:38:09.192828 13290 raft_consensus.cc:2454] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d80a4f92088429ba5880ad6dc299f5b in term 0.
I20260620 04:38:09.192976 13548 raft_consensus.cc:2454] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d80a4f92088429ba5880ad6dc299f5b in term 0.
I20260620 04:38:09.193365 13366 leader_election.cc:302] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d80a4f92088429ba5880ad6dc299f5b, 54e507948f904727844ec86bfbaac733; no voters:
I20260620 04:38:09.193670 13654 raft_consensus.cc:2781] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:09.193745 13654 raft_consensus.cc:482] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:09.193800 13654 raft_consensus.cc:3037] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:09.194941 13654 raft_consensus.cc:504] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:09.195346 13654 leader_election.cc:288] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [CANDIDATE]: Term 1 election: Requested vote from peers 54e507948f904727844ec86bfbaac733 (127.0.98.1:43785), 79429f37ac744ae88404babbe6468fbd (127.0.98.3:39427)
I20260620 04:38:09.195705 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4092a06d0364d3c9f13a46310acfbff" candidate_uuid: "1d80a4f92088429ba5880ad6dc299f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e507948f904727844ec86bfbaac733"
I20260620 04:38:09.195845 13290 raft_consensus.cc:3037] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:09.195749 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4092a06d0364d3c9f13a46310acfbff" candidate_uuid: "1d80a4f92088429ba5880ad6dc299f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79429f37ac744ae88404babbe6468fbd"
I20260620 04:38:09.196004 13548 raft_consensus.cc:3037] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:09.197144 13290 raft_consensus.cc:2454] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d80a4f92088429ba5880ad6dc299f5b in term 1.
I20260620 04:38:09.197185 13548 raft_consensus.cc:2454] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d80a4f92088429ba5880ad6dc299f5b in term 1.
I20260620 04:38:09.197427 13366 leader_election.cc:302] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d80a4f92088429ba5880ad6dc299f5b, 54e507948f904727844ec86bfbaac733; no voters:
I20260620 04:38:09.197628 13654 raft_consensus.cc:2781] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:09.197844 13654 raft_consensus.cc:686] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 1 LEADER]: Becoming Leader. State: Replica: 1d80a4f92088429ba5880ad6dc299f5b, State: Running, Role: LEADER
I20260620 04:38:09.198191 13654 consensus_queue.cc:237] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } }
I20260620 04:38:09.201364 13144 catalog_manager.cc:5495] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d80a4f92088429ba5880ad6dc299f5b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1d80a4f92088429ba5880ad6dc299f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d80a4f92088429ba5880ad6dc299f5b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41897 } health_report { overall_health: HEALTHY } } }
I20260620 04:38:09.269932 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.265940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.98.2:41897)
0620 04:38:09.266083 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:09.266089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:09.266166 (+ 77us) server_negotiation.cc:407] Connection header received
0620 04:38:09.266410 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:09.266413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:09.266483 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:09.266592 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:09.267016 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.267523 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.268430 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:09.268649 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:09.268765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:09.269072 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:09.269513 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:09.269645 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:09.269707 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 04:38:09.269762 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:38:09.276433 13290 raft_consensus.cc:1261] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 1 FOLLOWER]: Refusing update from remote peer 1d80a4f92088429ba5880ad6dc299f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:09.276451 13548 raft_consensus.cc:1261] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 1 FOLLOWER]: Refusing update from remote peer 1d80a4f92088429ba5880ad6dc299f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:09.277155 13654 consensus_queue.cc:1035] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e507948f904727844ec86bfbaac733" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:09.277364 13664 consensus_queue.cc:1035] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79429f37ac744ae88404babbe6468fbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:09.286334 13668 mvcc.cc:205] Tried to move back new op lower bound from 7298786464868720640 to 7298786464558387200. Current Snapshot: MvccSnapshot[applied={T|T < 7298786464868720640}]
I20260620 04:38:09.286681 13669 mvcc.cc:205] Tried to move back new op lower bound from 7298786464868720640 to 7298786464558387200. Current Snapshot: MvccSnapshot[applied={T|T < 7298786464868720640}]
I20260620 04:38:09.295096 13142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:09.295295 13142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:09.298123 13142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0a9a323c2c84f3cb2a0fe63c40f7856: Sending DeleteTablet for 3 replicas of tablet e4092a06d0364d3c9f13a46310acfbff
I20260620 04:38:09.298898 13544 tablet_service.cc:1514] Processing DeleteTablet for tablet e4092a06d0364d3c9f13a46310acfbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51116
I20260620 04:38:09.298902 13259 tablet_service.cc:1514] Processing DeleteTablet for tablet e4092a06d0364d3c9f13a46310acfbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40816
I20260620 04:38:09.299435 13676 tablet_replica.cc:330] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733: stopping tablet replica
I20260620 04:38:09.299669 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet e4092a06d0364d3c9f13a46310acfbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43984
I20260620 04:38:09.299894 13676 raft_consensus.cc:2229] T e4092a06d0364d3c9f13a46310acfbff P 54e507948f904727844ec86bfbaac733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:09.300320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13182
I20260620 04:38:09.301599 13678 tablet_replica.cc:330] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: stopping tablet replica
I20260620 04:38:09.302114 13678 raft_consensus.cc:2229] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:38:09.300168 13677 tablet_replica.cc:330] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: stopping tablet replica
I20260620 04:38:09.302711 13678 raft_consensus.cc:2258] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:09.302821 13677 raft_consensus.cc:2229] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:09.303133 13677 raft_consensus.cc:2258] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:09.304430 13678 ts_tablet_manager.cc:1905] T e4092a06d0364d3c9f13a46310acfbff P 1d80a4f92088429ba5880ad6dc299f5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:09.304610 13677 ts_tablet_manager.cc:1905] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:38:09.307600 13131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43785 (error 108)
I20260620 04:38:09.307765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13346
I20260620 04:38:09.307327 13677 ts_tablet_manager.cc:1918] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:09.308135 13677 log.cc:1196] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/e4092a06d0364d3c9f13a46310acfbff
I20260620 04:38:09.308634 13677 ts_tablet_manager.cc:1939] T e4092a06d0364d3c9f13a46310acfbff P 79429f37ac744ae88404babbe6468fbd: Deleting consensus metadata
I20260620 04:38:09.309806 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.309530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43785 (local address 127.0.0.1:40828)
0620 04:38:09.309644 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:09.309715 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 04:38:09.309875 13128 catalog_manager.cc:4841] TS 79429f37ac744ae88404babbe6468fbd (127.0.98.3:39427): tablet e4092a06d0364d3c9f13a46310acfbff (table test-table [id=3d43bc8e97524c96babd1ea53380e944]) successfully deleted
W20260620 04:38:09.309921 13131 catalog_manager.cc:4584] TS 54e507948f904727844ec86bfbaac733 (127.0.98.1:43785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4092a06d0364d3c9f13a46310acfbff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43785: connect: Connection refused (error 111)
I20260620 04:38:09.314862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13479
I20260620 04:38:09.315611 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:09.315013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41897 (local address 127.0.0.1:43998)
0620 04:38:09.315446 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:09.315509 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260620 04:38:09.320727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13111
I20260620 04:38:09.327436 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:38:09.672554 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:38:09.672698 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:38:10.018637 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:38:10.018 UTC [12178] LOG: received smart shutdown request
2026-06-20 04:38:10.022 UTC [12178] LOG: background worker "logical replication launcher" (PID 12187) exited with exit code 1
2026-06-20 04:38:10.022 UTC [12182] LOG: shutting down
2026-06-20 04:38:10.229 UTC [12178] LOG: database system is shut down
2026-06-20T04:38:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50559 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:38:10 dist-test-slave-zmf4 krb5kdc[13688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:38:10 dist-test-slave-zmf4 krb5kdc[13688](info): set up 2 sockets
Jun 20 04:38:10 dist-test-slave-zmf4 krb5kdc[13688](info): commencing operation
krb5kdc: starting...
W20260620 04:38:11.460320 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:38:11 dist-test-slave-zmf4 krb5kdc[13688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:38:11.585242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:11.693866 13701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:11.694185 13701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:11.694263 13701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:11.698251 13701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:11.698343 13701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:11.698383 13701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:11.698421 13701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:11.698455 13701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:11.703495 13701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:11.704535 13701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:11.705514 13701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:11.713688 13709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:11.713872 13701 server_base.cc:1034] running on GCE node
W20260620 04:38:11.713694 13707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:11.713694 13706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:11.714521 13701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:11.715489 13701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:11.716674 13701 hybrid_clock.cc:648] HybridClock initialized: now 1781930291716644 us; error 41 us; skew 500 ppm
Jun 20 04:38:11 dist-test-slave-zmf4 krb5kdc[13688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:11.719424 13701 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:11.720332 13701 webserver.cc:460] Webserver started at http://127.0.98.62:35465/ using document root <none> and password file <none>
I20260620 04:38:11.720901 13701 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:11.720992 13701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:11.721216 13701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:11.723174 13701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "92877beaedd240f7861c3f1da18980dd"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "65e01cf39b480133a80aefc793beea0e"
server_key_iv: "87c517c3650b9dde2cfa25b5041409d0"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.723728 13701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "92877beaedd240f7861c3f1da18980dd"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "65e01cf39b480133a80aefc793beea0e"
server_key_iv: "87c517c3650b9dde2cfa25b5041409d0"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.727284 13701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:38:11.729727 13716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:11.730913 13701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:38:11.731074 13701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "92877beaedd240f7861c3f1da18980dd"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "65e01cf39b480133a80aefc793beea0e"
server_key_iv: "87c517c3650b9dde2cfa25b5041409d0"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.731186 13701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:11.761431 13701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:11.764683 13701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:11.764887 13701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:11.773066 13701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42997
I20260620 04:38:11.773129 13768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42997 every 8 connection(s)
I20260620 04:38:11.774133 13701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:11.776908 13769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:11.780604 13769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:11.781635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13701
I20260620 04:38:11.781752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:38:11.782015 392 external_mini_cluster.cc:1428] Setting key 4fca36d9b1622b198220c5edb994c024
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:38:11 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:11.800405 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.783070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42997 (local address 127.0.0.1:56808)
0620 04:38:11.783561 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.783598 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:11.784823 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:11.785701 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:11.785706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:11.786064 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:11.786340 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:11.786353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:11.788179 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:11.788182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:11.788959 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:11.788969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:11.789889 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:11.789893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:11.790002 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:11.790634 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:11.790651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:11.794518 (+ 3867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:11.797306 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:11.797314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:11.797321 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:11.797561 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:11.798041 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:11.798045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:11.798047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:11.798196 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:11.799000 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:11.799006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:11.799142 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 04:38:11.799295 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 04:38:11.799546 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":142,"threads_started":1}
I20260620 04:38:11.800421 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.782993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.0.98.62:42997)
0620 04:38:11.783398 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:11.783418 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:11.783703 (+ 285us) server_negotiation.cc:407] Connection header received
0620 04:38:11.785228 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:11.785236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:11.785460 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:11.785651 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:11.786508 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:11.788002 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:11.789091 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:11.789710 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:11.789895 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:11.794675 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:11.794680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:11.794690 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:11.794711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:11.797125 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:11.797710 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:11.797720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:11.797725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:11.797883 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:11.798331 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:11.798336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:11.798340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:11.798868 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:11.798984 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:11.799433 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 04:38:11.799663 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260620 04:38:11.802937 13769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:11.803665 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.803389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55388)
0620 04:38:11.803529 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.803610 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 04:38:11.803943 13719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
I20260620 04:38:11.804932 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.804678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55398)
0620 04:38:11.804820 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.804874 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 04:38:11.805128 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43647: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:38:11.812428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43647
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:11.854431 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attempt: 1
I20260620 04:38:11.855338 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.855037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55414)
0620 04:38:11.855190 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.855255 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:38:11.856115 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.855889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55430)
0620 04:38:11.856010 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.856058 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 04:38:11.856297 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43647: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
W20260620 04:38:11.923952 13776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:11.924227 13776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:11.924268 13776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:38:11.924557 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attempt: 2
I20260620 04:38:11.925458 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.925130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55434)
0620 04:38:11.925321 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.925384 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 04:38:11.926273 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:11.926038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55444)
0620 04:38:11.926164 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:11.926214 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 04:38:11.926450 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43647: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43647: connect: Connection refused (error 111)
W20260620 04:38:11.929188 13776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:11.929260 13776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:11.929282 13776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:11.929298 13776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:11.929314 13776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:11.935138 13776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:11.936266 13776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:11.937235 13776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:11.945569 13782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:11.945590 13784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:11.945652 13776 server_base.cc:1034] running on GCE node
W20260620 04:38:11.945569 13781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:11.946389 13776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:11.947417 13776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:11.948633 13776 hybrid_clock.cc:648] HybridClock initialized: now 1781930291948619 us; error 25 us; skew 500 ppm
Jun 20 04:38:11 dist-test-slave-zmf4 krb5kdc[13688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:11.951403 13776 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:38:11.952301 13776 webserver.cc:460] Webserver started at http://127.0.98.61:37951/ using document root <none> and password file <none>
I20260620 04:38:11.952848 13776 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:11.952935 13776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:11.953157 13776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:11.955147 13776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data/instance:
uuid: "615e5b5c4f024109880d2e54cb416e85"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "b84c0f571aee190df96ec5ec5c192977"
server_key_iv: "895601780d5bd8721b224b09fc94781b"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.955693 13776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal/instance:
uuid: "615e5b5c4f024109880d2e54cb416e85"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "b84c0f571aee190df96ec5ec5c192977"
server_key_iv: "895601780d5bd8721b224b09fc94781b"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.959357 13776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:38:11.961769 13791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:11.963102 13776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:38:11.963261 13776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "615e5b5c4f024109880d2e54cb416e85"
format_stamp: "Formatted at 2026-06-20 04:38:11 on dist-test-slave-zmf4"
server_key: "b84c0f571aee190df96ec5ec5c192977"
server_key_iv: "895601780d5bd8721b224b09fc94781b"
server_key_version: "encryptionkey@0"
I20260620 04:38:11.963383 13776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:11.996630 13776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:12.000020 13776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:12.000263 13776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:12.007735 13776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43647
I20260620 04:38:12.007762 13843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43647 every 8 connection(s)
I20260620 04:38:12.008780 13776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:38:12.008981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13776
I20260620 04:38:12.009089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-1/wal/instance
I20260620 04:38:12.009562 392 external_mini_cluster.cc:1428] Setting key 9266257d30c43327d344efc67633035d
I20260620 04:38:12.012117 13844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:12.015674 13844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 04:38:12.017607 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:12.030742 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.017959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55450)
0620 04:38:12.018068 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.018083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.018177 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.019722 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.019725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.019937 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.020063 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.020070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.022380 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.022383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.022972 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.022980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.023740 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.023743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.023854 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.024292 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.024307 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.025875 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.028543 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.028546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.028548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.028823 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.029268 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.029274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.029278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.029389 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.030227 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.030230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.030313 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.030431 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.030567 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":23}
I20260620 04:38:12.030879 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.017600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.62:42997)
0620 04:38:12.017755 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.017760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.018633 (+ 873us) server_negotiation.cc:407] Connection header received
0620 04:38:12.019546 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.019549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.019594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.019677 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.020798 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.021458 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.022554 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.023048 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.023167 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.026824 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.026828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.026831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.026861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.028568 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.029088 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.029092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.029093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.029206 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.029544 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.029547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.029549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.029934 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.030054 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.030518 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.030662 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":58}
I20260620 04:38:12.031651 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.018077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.0.98.61:43647)
0620 04:38:12.018477 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.018495 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.018533 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:38:12.019253 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.019270 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.019560 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.020237 (+ 677us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.020557 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.021833 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.023088 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.023581 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.023737 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.026017 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.026028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.026037 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.026058 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.028417 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.028971 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.028975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.028976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.029163 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.029541 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.029544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.029546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.030119 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.030214 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.030886 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.031027 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"spinlock_wait_cycles":17152,"thread_start_us":148,"threads_started":1}
I20260620 04:38:12.031647 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.018056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42997 (local address 127.0.0.1:56812)
0620 04:38:12.018511 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.018552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.019305 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.019781 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.019793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.020139 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.020662 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.020672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.021580 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.021584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.022431 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.022439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.023178 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.023181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.023374 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.024065 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.024080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.026680 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.028675 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.028683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.028692 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.028985 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.029297 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.029300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.029301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.029436 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.030080 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.030085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.030261 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.030448 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.030727 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260620 04:38:12.033077 13844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.033783 13769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.034461 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.034224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53268)
0620 04:38:12.034345 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.034398 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 04:38:12.035219 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.034983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53284)
0620 04:38:12.035120 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.035172 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 04:38:12.035370 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:38:12.039073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33229
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:12.042527 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.033803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.98.61:43647)
0620 04:38:12.033911 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.033915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.033928 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:38:12.034008 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.034010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.034051 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.034131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.034608 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.035329 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.036304 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.036646 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.036734 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.038919 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.038922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.038924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.038942 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.040518 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.041045 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.041048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.041049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.041162 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.041534 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.041537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.041538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.041728 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.041819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.042151 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.042395 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":51}
I20260620 04:38:12.042542 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.033673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55454)
0620 04:38:12.033826 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.033840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.033917 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.034209 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.034211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.034390 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.034482 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.034491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.035435 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.035440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.036142 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.036152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.036777 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.036780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.036865 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.037368 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.037380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.038776 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.040634 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.040641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.040644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.040919 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.041266 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.041269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.041274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.041409 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.041842 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.041848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.041964 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.042084 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.042264 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 04:38:12.043560 13844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.044471 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.044078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53286)
0620 04:38:12.044319 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.044409 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 04:38:12.045327 13792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
I20260620 04:38:12.046588 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.046329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53292)
0620 04:38:12.046484 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.046533 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 04:38:12.046973 13844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
I20260620 04:38:12.066630 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 1
I20260620 04:38:12.067591 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.067234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53300)
0620 04:38:12.067442 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.067517 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 04:38:12.068396 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.068144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53306)
0620 04:38:12.068282 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.068342 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 04:38:12.068522 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
I20260620 04:38:12.096246 13844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 1
I20260620 04:38:12.097195 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.096875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53322)
0620 04:38:12.097052 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.097115 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 04:38:12.097930 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.097707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53332)
0620 04:38:12.097832 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.097875 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 04:38:12.098042 13844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
I20260620 04:38:12.143702 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 2
I20260620 04:38:12.144591 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.144276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53348)
0620 04:38:12.144453 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.144516 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 04:38:12.145406 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.145177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53350)
0620 04:38:12.145299 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.145352 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 04:38:12.145540 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
W20260620 04:38:12.152381 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:12.152631 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:12.152710 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:12.156682 13850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:12.156773 13850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:12.156818 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:12.156874 13850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:12.156910 13850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:12.162338 13850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42997,127.0.98.61:43647,127.0.98.60:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:12.163431 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:12.164489 13850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:38:12.166242 13844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 2
I20260620 04:38:12.167188 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.166864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53362)
0620 04:38:12.167049 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.167112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 04:38:12.167922 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.167717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53366)
0620 04:38:12.167821 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.167871 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:38:12.168057 13844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
W20260620 04:38:12.173998 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:12.174058 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:12.174100 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:12.174362 13850 server_base.cc:1034] running on GCE node
I20260620 04:38:12.174892 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:12.175973 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:12.177177 13850 hybrid_clock.cc:648] HybridClock initialized: now 1781930292177169 us; error 67 us; skew 500 ppm
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:12.179992 13850 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:38:12.180905 13850 webserver.cc:460] Webserver started at http://127.0.98.60:39239/ using document root <none> and password file <none>
I20260620 04:38:12.181457 13850 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:12.181546 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:12.181770 13850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:12.183720 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data/instance:
uuid: "c0386f5253b14ebab066405b862945ad"
format_stamp: "Formatted at 2026-06-20 04:38:12 on dist-test-slave-zmf4"
server_key: "e05f7bebe7a9bd8ad85a28fdb0373a58"
server_key_iv: "f4cf5207c24e1ae39fe7181cbcda477f"
server_key_version: "encryptionkey@0"
I20260620 04:38:12.184319 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal/instance:
uuid: "c0386f5253b14ebab066405b862945ad"
format_stamp: "Formatted at 2026-06-20 04:38:12 on dist-test-slave-zmf4"
server_key: "e05f7bebe7a9bd8ad85a28fdb0373a58"
server_key_iv: "f4cf5207c24e1ae39fe7181cbcda477f"
server_key_version: "encryptionkey@0"
I20260620 04:38:12.188158 13850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 04:38:12.190621 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:12.192111 13850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:12.192305 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "c0386f5253b14ebab066405b862945ad"
format_stamp: "Formatted at 2026-06-20 04:38:12 on dist-test-slave-zmf4"
server_key: "e05f7bebe7a9bd8ad85a28fdb0373a58"
server_key_iv: "f4cf5207c24e1ae39fe7181cbcda477f"
server_key_version: "encryptionkey@0"
I20260620 04:38:12.192430 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:12.238648 13850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:12.241930 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:12.242161 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:12.244707 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 3
I20260620 04:38:12.245512 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.245205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53368)
0620 04:38:12.245382 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.245437 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 04:38:12.246343 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.246056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53380)
0620 04:38:12.246219 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.246280 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 04:38:12.246524 13769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33229: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33229: connect: Connection refused (error 111)
I20260620 04:38:12.249969 13850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33229
I20260620 04:38:12.250033 13918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33229 every 8 connection(s)
I20260620 04:38:12.250900 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:38:12.253747 13919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:12.256872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 13850
I20260620 04:38:12.257025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781930128116734-392-0/minicluster-data/master-2/wal/instance
I20260620 04:38:12.257280 392 external_mini_cluster.cc:1428] Setting key ca7551c1cd8397a0f27002d79a1d1072
I20260620 04:38:12.257565 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.259250 13844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 3
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:38:12.271639 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.259399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.62:42997)
0620 04:38:12.259528 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.259535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.259923 (+ 388us) server_negotiation.cc:407] Connection header received
0620 04:38:12.260352 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.260356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.260418 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.260505 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.261403 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.261890 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.263337 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.263668 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.263762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.267705 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.267709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.267711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.267736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.269572 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270166 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.270171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.270173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.270263 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270612 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.270616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.270618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.270865 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.270964 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.271355 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.271473 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:38:12.271729 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.259650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53390)
0620 04:38:12.259866 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.259882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.259956 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.260655 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.260660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.260841 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.260952 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.260959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.262714 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.262718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.263477 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.263506 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.264318 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.264322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.264421 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.264927 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.264946 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.266387 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.269888 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.269891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.269893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.270090 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.270461 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.270467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.270561 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.271176 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.271180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.271285 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.271404 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.271550 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 04:38:12.272523 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.259323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42997 (local address 127.0.0.1:56816)
0620 04:38:12.259781 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.259843 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.260128 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.260559 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.260567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.261024 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.261274 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.261283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.261994 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.261999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.263207 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.263218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.263804 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.263809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.264134 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.264769 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.264786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.267562 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.269706 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.269714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.269723 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.270061 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.270357 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.270363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.270501 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.270973 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.270978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.271101 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.271300 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.271590 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":232,"thread_start_us":155,"threads_started":1}
I20260620 04:38:12.272552 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.260079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.0.98.60:33229)
0620 04:38:12.260406 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.260415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.260426 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:38:12.260455 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.260458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.260499 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.260685 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.261152 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.262589 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.263600 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.264158 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.264307 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.266708 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.266715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.266732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.266752 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.269778 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270203 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.270210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.270213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.270362 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.270637 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.270643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.270645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.271047 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.271163 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.272089 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.272351 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":358,"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260620 04:38:12.273552 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.258513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.0.98.60:33229)
0620 04:38:12.258897 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.258919 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.258975 (+ 56us) server_negotiation.cc:407] Connection header received
0620 04:38:12.259840 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.259860 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.260143 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.261266 (+ 1123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.261410 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.262643 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.263546 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.264166 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.264680 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.267740 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.267744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.267747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.267774 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.270837 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.271356 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.271362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.271367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.271445 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.272351 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.272354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.272355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.272556 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.272717 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.272943 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.273379 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":252,"thread_start_us":127,"threads_started":1}
I20260620 04:38:12.273918 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.277144 13844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: Bootstrap starting.
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:38:12.280017 13844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: Neither blocks nor log segments found. Creating new log.
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:12.280966 13844 log.cc:824] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:12.283460 13844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: No bootstrap required, opened a new log
I20260620 04:38:12.283568 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.274385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43647 (local address 127.0.0.1:55470)
0620 04:38:12.274645 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.274660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.274734 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.275081 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.275083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.275209 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.275298 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.275302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.276234 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.276239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.277024 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.277033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.277685 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.277687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.277761 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.278573 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.278584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.279862 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.281856 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.281859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.281860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.282036 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.282433 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.282436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.282438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.282553 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.283037 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.283040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.283137 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.283254 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.283403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 04:38:12.283913 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.274620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.98.61:43647)
0620 04:38:12.274726 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.274730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.274743 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:38:12.274848 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.274851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.274893 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.274944 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.275448 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.276083 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.277148 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.277436 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.277492 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.279990 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.279994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.279996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.280026 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.281714 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.282140 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.282145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.282147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.282231 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.282651 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.282658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.282662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.282910 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.282994 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.283595 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.283730 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:38:12.284590 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:38:12.286662 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.275450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.0.98.62:42997)
0620 04:38:12.275561 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.275565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.275576 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:38:12.275608 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.275611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.275652 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.275729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.276481 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.277183 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.278169 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.278396 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.278474 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.282199 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.282203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.282205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.282229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.284667 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.285210 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.285214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.285215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.285299 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.285647 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.285650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.285652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.285882 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.286089 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.286265 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.286393 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:38:12.287762 13844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.288251 13844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:12.288313 13844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615e5b5c4f024109880d2e54cb416e85, State: Initialized, Role: FOLLOWER
I20260620 04:38:12.288976 13844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:38:12.290485 13844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:12.292496 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.292661 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.293568 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.285080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53396)
0620 04:38:12.285215 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.285227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.285314 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.285516 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.285519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.285677 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.285737 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.285741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.286681 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.286684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.287272 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.287280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.287799 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.287802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.287922 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.288457 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.288470 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.289648 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.291434 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.291441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.291445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.291704 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.292302 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.292305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.292307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.292401 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.292933 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.292937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.293016 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.293094 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.293378 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":32}
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:38:12.294229 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.285101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.0.98.60:33229)
0620 04:38:12.285197 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.285201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.285278 (+ 77us) server_negotiation.cc:407] Connection header received
0620 04:38:12.285381 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.285385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.285417 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.285509 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.285842 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.286571 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.287414 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.287688 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.287802 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.289783 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.289786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.289789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.289810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.291309 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.292044 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.292047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.292048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.292159 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.292514 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.292518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.292523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.292818 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.293861 (+ 1043us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.293933 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.294051 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":31}
W20260620 04:38:12.296473 13937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:38:12.296543 13937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:38:12.294186 13776 master_runner.cc:186] Error getting master registration for 127.0.98.62:42997: OK, instance_id { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" instance_seqno: 1781930291770748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:38:12.297518 13776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:38:12.298038 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: Bootstrap starting.
I20260620 04:38:12.298868 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.287918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.0.98.61:43647)
0620 04:38:12.288025 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.288030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.288044 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:38:12.288151 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.288155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.288243 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.288322 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.289344 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.290041 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.291042 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.291424 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.291516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.295186 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.295189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.295193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.295241 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.296970 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.297582 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.297585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.297587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.297668 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.298000 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.298003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.298005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.298230 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.298406 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.298619 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.298728 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:38:12.300688 13919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:12.301405 13919 log.cc:824] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:12.302155 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:38:12.303642 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: No bootstrap required, opened a new log
I20260620 04:38:12.307672 13919 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.307925 13919 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:12.307991 13919 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0386f5253b14ebab066405b862945ad, State: Initialized, Role: FOLLOWER
I20260620 04:38:12.308475 13919 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.310030 13919 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:12.312594 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.312794 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.314538 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.305058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.98.62:42997)
0620 04:38:12.305167 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.305172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.305186 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:38:12.305460 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.305463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.305515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.305600 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.306449 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.307227 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.308370 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.308712 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.308824 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.310566 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.310570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.310573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.310599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.312312 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.312784 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.312787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.312789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.312874 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.313178 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.313183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.313185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.313418 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.313544 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.314195 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.314333 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:38:12.313652 13850 master_runner.cc:186] Error getting master registration for 127.0.98.62:42997: OK, instance_id { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" instance_seqno: 1781930291770748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:38:12.318620 13850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:38:12.321964 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.304706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.98.60:33229)
0620 04:38:12.304804 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.304809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.312436 (+ 7627us) server_negotiation.cc:407] Connection header received
0620 04:38:12.312618 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.312623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.312672 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.312726 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.313694 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.314494 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.316290 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.316620 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.316697 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.318585 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.318589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.318591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.318615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.320347 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.320761 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.320764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.320766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.320846 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.321126 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.321129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.321131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.321300 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.321452 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.321673 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.321786 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:38:12.322371 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.305069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.0.98.61:43647)
0620 04:38:12.305173 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.305177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.313700 (+ 8523us) server_negotiation.cc:407] Connection header received
0620 04:38:12.313881 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.313885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.313940 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.313993 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.314918 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.315655 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.317002 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.317338 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.317407 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.319011 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.319015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.319018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.319043 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.320655 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.321143 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.321146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.321148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.321224 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.321509 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.321512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.321514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.321701 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.321799 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.322116 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.322224 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 04:38:12.326604 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:38:12.326696 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:38:12.410696 13769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } attempt: 4
Jun 20 04:38:12 dist-test-slave-zmf4 krb5kdc[13688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:38:12.421324 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.411314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.0.98.60:33229)
0620 04:38:12.411496 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:12.411502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:12.411523 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:38:12.411659 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:12.411662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:12.411708 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:12.411791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:12.412307 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.413104 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.414228 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.414547 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.414658 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.417141 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:12.417145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:12.417148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:12.417174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:12.419423 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.419937 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.419941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.419943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.420040 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:12.420391 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:12.420394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:12.420396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:12.420622 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:12.420732 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:12.421052 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 04:38:12.421176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 04:38:12.421249 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:12.411258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33229 (local address 127.0.0.1:53416)
0620 04:38:12.411452 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:12.411467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:12.411551 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:12.411824 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:12.411827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:12.412061 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:12.412169 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.412176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.413234 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.413237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.414087 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:12.414098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:12.414681 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:12.414684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:12.414772 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:12.415342 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:12.415358 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:12.416993 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:12.419544 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.419548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.419550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.419801 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.420150 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:12.420153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:12.420155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:12.420284 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:12.420735 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:12.420738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:12.420843 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 04:38:12.420952 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 04:38:12.421090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 04:38:12.425223 13769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: Bootstrap starting.
I20260620 04:38:12.428470 13769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:12.429450 13769 log.cc:824] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:12.431653 13769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: No bootstrap required, opened a new log
I20260620 04:38:12.436421 13769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.436674 13769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:12.436728 13769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92877beaedd240f7861c3f1da18980dd, State: Initialized, Role: FOLLOWER
I20260620 04:38:12.438449 13769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.439750 13769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:12.440783 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.440990 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: This master's current role is: FOLLOWER
W20260620 04:38:12.444593 13972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:38:12.444716 13972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:38:12.447623 13701 master_runner.cc:263] No leader master found from master 92877beaedd240f7861c3f1da18980dd
I20260620 04:38:12.447681 13701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 04:38:12.493714 13872 tablet.cc:2369] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:38:12.502018 13797 tablet.cc:2369] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:12.502427 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:12.502664 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.503592 13947 leader_election.cc:288] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92877beaedd240f7861c3f1da18980dd (127.0.98.62:42997), 615e5b5c4f024109880d2e54cb416e85 (127.0.98.61:43647)
I20260620 04:38:12.504136 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0386f5253b14ebab066405b862945ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615e5b5c4f024109880d2e54cb416e85" is_pre_election: true
I20260620 04:38:12.504227 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0386f5253b14ebab066405b862945ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92877beaedd240f7861c3f1da18980dd" is_pre_election: true
I20260620 04:38:12.504482 13818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0386f5253b14ebab066405b862945ad in term 0.
I20260620 04:38:12.504527 13743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0386f5253b14ebab066405b862945ad in term 0.
I20260620 04:38:12.504920 13869 leader_election.cc:302] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615e5b5c4f024109880d2e54cb416e85, c0386f5253b14ebab066405b862945ad; no voters:
I20260620 04:38:12.505187 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:12.505275 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:12.505324 13947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:12.506317 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.506675 13947 leader_election.cc:288] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [CANDIDATE]: Term 1 election: Requested vote from peers 92877beaedd240f7861c3f1da18980dd (127.0.98.62:42997), 615e5b5c4f024109880d2e54cb416e85 (127.0.98.61:43647)
I20260620 04:38:12.507018 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0386f5253b14ebab066405b862945ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615e5b5c4f024109880d2e54cb416e85"
I20260620 04:38:12.507122 13818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:12.508119 13818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0386f5253b14ebab066405b862945ad in term 1.
I20260620 04:38:12.508373 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0386f5253b14ebab066405b862945ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92877beaedd240f7861c3f1da18980dd"
I20260620 04:38:12.508424 13869 leader_election.cc:302] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615e5b5c4f024109880d2e54cb416e85, c0386f5253b14ebab066405b862945ad; no voters:
I20260620 04:38:12.508504 13743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:12.508577 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:12.508805 13947 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [term 1 LEADER]: Becoming Leader. State: Replica: c0386f5253b14ebab066405b862945ad, State: Running, Role: LEADER
I20260620 04:38:12.509137 13947 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } }
I20260620 04:38:12.509687 13743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0386f5253b14ebab066405b862945ad in term 1.
I20260620 04:38:12.511195 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0386f5253b14ebab066405b862945ad. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.511387 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:12.512447 13976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:12.513301 13976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 04:38:12.516673 13722 tablet.cc:2369] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:12.517854 13743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [term 1 FOLLOWER]: Refusing update from remote peer c0386f5253b14ebab066405b862945ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:12.517982 13818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [term 1 FOLLOWER]: Refusing update from remote peer c0386f5253b14ebab066405b862945ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:12.518519 13947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:12.518741 13974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:12.522359 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0386f5253b14ebab066405b862945ad. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.522482 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.522419 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0386f5253b14ebab066405b862945ad. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.522526 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.524729 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.524847 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:12.524927 13977 mvcc.cc:205] Tried to move back new op lower bound from 7298786478147084288 to 7298786478120239104. Current Snapshot: MvccSnapshot[applied={T|T < 7298786478147084288}]
I20260620 04:38:12.525179 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.525292 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.525687 13979 mvcc.cc:205] Tried to move back new op lower bound from 7298786478147084288 to 7298786478120239104. Current Snapshot: MvccSnapshot[applied={T|T < 7298786478147084288}]
I20260620 04:38:12.525688 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.525789 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:12.526522 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0386f5253b14ebab066405b862945ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92877beaedd240f7861c3f1da18980dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42997 } } peers { permanent_uuid: "615e5b5c4f024109880d2e54cb416e85" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43647 } } peers { permanent_uuid: "c0386f5253b14ebab066405b862945ad" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33229 } } }
I20260620 04:38:12.526621 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:38:12.527216 13976 catalog_manager.cc:1347] Generated new cluster ID: 83cd488bfcd14079ab2bfb716702fc2c
I20260620 04:38:12.527268 13976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:12.544212 13976 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:38:12.545320 13976 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:12.562172 13976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0386f5253b14ebab066405b862945ad: Generated new TSK 0
I20260620 04:38:12.563356 13976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:38:13.297921 13937 catalog_manager.cc:1259] Loaded cluster ID: 83cd488bfcd14079ab2bfb716702fc2c
I20260620 04:38:13.298008 13937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: loading cluster ID for follower catalog manager: success
I20260620 04:38:13.299589 13937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: acquiring CA information for follower catalog manager: success
I20260620 04:38:13.300716 13937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 615e5b5c4f024109880d2e54cb416e85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:38:13.446018 13972 catalog_manager.cc:1259] Loaded cluster ID: 83cd488bfcd14079ab2bfb716702fc2c
I20260620 04:38:13.446096 13972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: loading cluster ID for follower catalog manager: success
I20260620 04:38:13.447793 13972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: acquiring CA information for follower catalog manager: success
I20260620 04:38:13.449043 13972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92877beaedd240f7861c3f1da18980dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:38:13.632973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13701
I20260620 04:38:13.639056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13776
I20260620 04:38:13.644677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 13850
2026-06-20T04:38:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3291 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:38:13 dist-test-slave-zmf4 krb5kdc[13991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:38:13 dist-test-slave-zmf4 krb5kdc[13991](info): set up 2 sockets
Jun 20 04:38:13 dist-test-slave-zmf4 krb5kdc[13991](info): commencing operation
krb5kdc: starting...
W20260620 04:38:14.708734 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:38:14 dist-test-slave-zmf4 krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:38:14.835762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:14.945654 14004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:14.945938 14004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:14.945983 14004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:14.950084 14004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:14.950151 14004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:14.950171 14004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:14.950188 14004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:14.950203 14004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:14.956290 14004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:14.957389 14004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:14.958353 14004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:14.967376 14009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:14.967492 14004 server_base.cc:1034] running on GCE node
W20260620 04:38:14.967344 14012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:14.967376 14010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:14.968727 14004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:14.969975 14004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:14.971313 14004 hybrid_clock.cc:648] HybridClock initialized: now 1781930294971273 us; error 34 us; skew 500 ppm
Jun 20 04:38:14 dist-test-slave-zmf4 krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:14.974781 14004 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:14.975829 14004 webserver.cc:460] Webserver started at http://127.0.98.62:36355/ using document root <none> and password file <none>
I20260620 04:38:14.976550 14004 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:14.976645 14004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:14.976895 14004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:14.978940 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "634de6409f7142ddb988c5eec77d6682"
format_stamp: "Formatted at 2026-06-20 04:38:14 on dist-test-slave-zmf4"
server_key: "a94dda6c730b09332858d0e3b735dd88"
server_key_iv: "98a93c86014d5125388a587e8f5a45e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:14.979509 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "634de6409f7142ddb988c5eec77d6682"
format_stamp: "Formatted at 2026-06-20 04:38:14 on dist-test-slave-zmf4"
server_key: "a94dda6c730b09332858d0e3b735dd88"
server_key_iv: "98a93c86014d5125388a587e8f5a45e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:14.983423 14004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:38:14.986701 14019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:14.988369 14004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:14.988539 14004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "634de6409f7142ddb988c5eec77d6682"
format_stamp: "Formatted at 2026-06-20 04:38:14 on dist-test-slave-zmf4"
server_key: "a94dda6c730b09332858d0e3b735dd88"
server_key_iv: "98a93c86014d5125388a587e8f5a45e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:14.988673 14004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:14.999540 14004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:15.002930 14004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:15.003150 14004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:15.011713 14004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44357
I20260620 04:38:15.011734 14071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44357 every 8 connection(s)
I20260620 04:38:15.013273 14004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:15.016364 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:15.021867 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682: Bootstrap starting.
I20260620 04:38:15.022357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14004
I20260620 04:38:15.022472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:38:15.022732 392 external_mini_cluster.cc:1428] Setting key 8367f046592123190272fac99d1ff7a2
I20260620 04:38:15.024561 14072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:15.025290 14072 log.cc:824] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:15.027376 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682: No bootstrap required, opened a new log
Jun 20 04:38:15 dist-test-slave-zmf4 krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:15.031697 14072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260620 04:38:15.032015 14072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:15.032115 14072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634de6409f7142ddb988c5eec77d6682, State: Initialized, Role: FOLLOWER
I20260620 04:38:15.032653 14072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260620 04:38:15.032827 14072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:15.032925 14072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:15.033031 14072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:15.033989 14072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260620 04:38:15.034360 14072 leader_election.cc:302] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634de6409f7142ddb988c5eec77d6682; no voters:
I20260620 04:38:15.034734 14072 leader_election.cc:288] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:38:15.034813 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:15.035028 14077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [term 1 LEADER]: Becoming Leader. State: Replica: 634de6409f7142ddb988c5eec77d6682, State: Running, Role: LEADER
I20260620 04:38:15.035354 14077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260620 04:38:15.036104 14072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:15.036609 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634de6409f7142ddb988c5eec77d6682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260620 04:38:15.036732 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:15.036976 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634de6409f7142ddb988c5eec77d6682. Latest consensus state: current_term: 1 leader_uuid: "634de6409f7142ddb988c5eec77d6682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634de6409f7142ddb988c5eec77d6682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260620 04:38:15.037086 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:15.037225 14081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:15.037995 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:15.024035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.98.62:44357)
0620 04:38:15.024491 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:15.024506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:15.024540 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:38:15.025450 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:15.025478 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:15.025813 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:15.026155 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:15.026946 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:15.027994 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:15.028901 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:15.029382 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:15.029568 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:15.032561 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:15.032566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:15.032580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:15.032606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:15.034542 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:15.035052 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:15.035058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:15.035070 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:15.035229 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:15.035540 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:15.035543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:15.035545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:15.035929 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:15.036138 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:15.036423 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 04:38:15.036833 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":165,"threads_started":1}
I20260620 04:38:15.040361 14081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:15.041792 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:38:15.046422 14081 catalog_manager.cc:1347] Generated new cluster ID: 99564fbb891d41dd98172aa184827d28
I20260620 04:38:15.046515 14081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:15.068989 14081 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:38:15.070096 14081 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:15.076231 14081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634de6409f7142ddb988c5eec77d6682: Generated new TSK 0
I20260620 04:38:15.077000 14081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:38:15.097832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14004
2026-06-20T04:38:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:38:15 dist-test-slave-zmf4 krb5kdc[14099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:38:15 dist-test-slave-zmf4 krb5kdc[14099](info): set up 2 sockets
Jun 20 04:38:15 dist-test-slave-zmf4 krb5kdc[14099](info): commencing operation
krb5kdc: starting...
W20260620 04:38:16.160655 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:38:16.287647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:16.407141 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:16.407442 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:16.407519 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:16.411742 14112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:16.411844 14112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:16.411895 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:16.411952 14112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:16.411996 14112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:16.417304 14112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:16.418324 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:16.419297 14112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:16.427744 14118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:16.427878 14112 server_base.cc:1034] running on GCE node
W20260620 04:38:16.427744 14117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:16.427798 14120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:16.428609 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:16.429641 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:16.430856 14112 hybrid_clock.cc:648] HybridClock initialized: now 1781930296430836 us; error 29 us; skew 500 ppm
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:16.437480 14112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:16.438406 14112 webserver.cc:460] Webserver started at http://127.0.98.62:35975/ using document root <none> and password file <none>
I20260620 04:38:16.438957 14112 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:16.439045 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:16.439272 14112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:16.441323 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "0df7d796ad894c448bc2e282258efe1c"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "87ee9d6a6203e05d6bba0fad1d6cebf9"
server_key_iv: "ba2ff1a7826ad92ab95d440eed9988e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.441876 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "0df7d796ad894c448bc2e282258efe1c"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "87ee9d6a6203e05d6bba0fad1d6cebf9"
server_key_iv: "ba2ff1a7826ad92ab95d440eed9988e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.445581 14112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:38:16.448040 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.449332 14112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:16.449456 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "0df7d796ad894c448bc2e282258efe1c"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "87ee9d6a6203e05d6bba0fad1d6cebf9"
server_key_iv: "ba2ff1a7826ad92ab95d440eed9988e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.449558 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:16.479856 14112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:16.483160 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:16.483366 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:16.490787 14112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42657
I20260620 04:38:16.491063 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42657 every 8 connection(s)
I20260620 04:38:16.491778 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:16.494508 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:16.494935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14112
I20260620 04:38:16.495062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:38:16.495369 392 external_mini_cluster.cc:1428] Setting key adc4b7404829ca77419025873746c1d3
I20260620 04:38:16.500762 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap starting.
I20260620 04:38:16.503319 14180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:16.504259 14180 log.cc:824] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:16.506645 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: No bootstrap required, opened a new log
I20260620 04:38:16.509969 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:16.496796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.98.62:42657)
0620 04:38:16.497190 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:16.497204 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:16.497238 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:38:16.498109 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:16.498126 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:16.498458 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:16.498752 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:16.499642 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.501038 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.502194 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.502663 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.502793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:16.505519 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:16.505524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:16.505537 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:16.505564 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:16.507303 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.507749 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.507775 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.507779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.507883 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.508166 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.508171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.508172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.508576 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:16.508786 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:16.509019 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 04:38:16.509300 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260620 04:38:16.510561 14180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:16.510814 14180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:16.510900 14180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Initialized, Role: FOLLOWER
I20260620 04:38:16.511381 14180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:16.511548 14180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:16.511607 14180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:16.511703 14180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:16.512830 14180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:16.513121 14180 leader_election.cc:302] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df7d796ad894c448bc2e282258efe1c; no voters:
I20260620 04:38:16.513382 14180 leader_election.cc:288] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:38:16.513531 14185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:16.513783 14185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 LEADER]: Becoming Leader. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Running, Role: LEADER
I20260620 04:38:16.514118 14185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:16.514573 14180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:16.515831 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df7d796ad894c448bc2e282258efe1c. Latest consensus state: current_term: 1 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:16.515874 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:16.515977 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:16.515980 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:16.516552 14194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:16.519598 14194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:16.525214 14194 catalog_manager.cc:1347] Generated new cluster ID: 43680da639494e369a96a6a1e6e5be3a
I20260620 04:38:16.525286 14194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:16.538197 14194 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:38:16.539093 14194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:16.543058 14194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Generated new TSK 0
I20260620 04:38:16.543658 14194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:38:16.575465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:16.687047 14206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:16.687310 14206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:16.687353 14206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:16.691498 14206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:16.691555 14206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:16.691614 14206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:16.697052 14206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:16.698132 14206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:16.699153 14206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:16.708931 14211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:16.709048 14212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:16.709197 14206 server_base.cc:1034] running on GCE node
W20260620 04:38:16.709229 14214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:16.709656 14206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:16.710289 14206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:16.711457 14206 hybrid_clock.cc:648] HybridClock initialized: now 1781930296711423 us; error 47 us; skew 500 ppm
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:16.714294 14206 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:16.715268 14206 webserver.cc:460] Webserver started at http://127.0.98.1:44509/ using document root <none> and password file <none>
I20260620 04:38:16.715811 14206 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:16.715898 14206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:16.716159 14206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:16.718164 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e11515aea16491baac8ab2ebe73e588"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "4c14643ee073f91a076180f3620936b5"
server_key_iv: "62b8ad234a3e1e56bb5f4428f363c6af"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.718663 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e11515aea16491baac8ab2ebe73e588"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "4c14643ee073f91a076180f3620936b5"
server_key_iv: "62b8ad234a3e1e56bb5f4428f363c6af"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.722302 14206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 04:38:16.724701 14221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.725950 14206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:16.726090 14206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "2e11515aea16491baac8ab2ebe73e588"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "4c14643ee073f91a076180f3620936b5"
server_key_iv: "62b8ad234a3e1e56bb5f4428f363c6af"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.726204 14206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:16.745910 14206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:16.749298 14206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:16.749507 14206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:16.750109 14206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:16.750932 14206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:16.751003 14206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.751068 14206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:16.751121 14206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.761062 14206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46275
I20260620 04:38:16.761082 14334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46275 every 8 connection(s)
I20260620 04:38:16.762214 14206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:16.771941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14206
I20260620 04:38:16.772102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:38:16.772469 392 external_mini_cluster.cc:1428] Setting key 663e4e14ca59d3302d4baad948231c9f
I20260620 04:38:16.776522 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:16.764174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53567 (local address 127.0.98.62:42657)
0620 04:38:16.764343 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:16.764346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:16.765364 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 04:38:16.766218 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:16.766221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:16.766269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:16.766347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:16.767500 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.768037 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.769230 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.769462 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.769552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:16.772141 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:16.772144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:16.772146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:16.772189 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:16.773974 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.774643 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.774646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.774647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.774746 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.775183 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.775186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.775187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.775380 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:16.775472 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:16.776142 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 04:38:16.776304 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 04:38:16.777364 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:16.764581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:53567)
0620 04:38:16.765221 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:16.765260 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:16.766000 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:16.766512 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:16.766518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:16.766860 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:16.767333 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:16.767345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.768214 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.768217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:16.769081 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:16.769090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.769725 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.769728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:16.769893 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:16.770512 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:16.770528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:16.771961 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:16.774143 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:16.774147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:16.774159 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:16.774528 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:16.774848 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:16.774854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:16.774859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:16.775049 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:16.775493 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:16.775502 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:16.775825 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 04:38:16.776065 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 04:38:16.776390 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":119,"threads_started":1}
I20260620 04:38:16.778683 14335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:16.778929 14335 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:16.779407 14335 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:16.780925 14144 ts_manager.cc:194] Registered new tserver with Master: 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1:46275)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:38:16.782070 14144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:38:16.799388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:16.909672 14340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:16.909927 14340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:16.909970 14340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:16.913975 14340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:16.914032 14340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:16.914120 14340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:16.919674 14340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:16.920781 14340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:16.921965 14340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:16.931915 14345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:16.931911 14348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:16.931911 14346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:16.931991 14340 server_base.cc:1034] running on GCE node
I20260620 04:38:16.932512 14340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:16.933120 14340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:16.934342 14340 hybrid_clock.cc:648] HybridClock initialized: now 1781930296934308 us; error 27 us; skew 500 ppm
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:16.937000 14340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:16.937798 14340 webserver.cc:460] Webserver started at http://127.0.98.2:39983/ using document root <none> and password file <none>
I20260620 04:38:16.938342 14340 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:16.938433 14340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:16.938651 14340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:16.940752 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "e8ed9aa998974ff78ec6023920c7b4de"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "f5ffe563d2fb4bc42accd55e9d6eb83a"
server_key_iv: "7b5f1365882065a91ed9a76617747f76"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.941298 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e8ed9aa998974ff78ec6023920c7b4de"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "f5ffe563d2fb4bc42accd55e9d6eb83a"
server_key_iv: "7b5f1365882065a91ed9a76617747f76"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.944890 14340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:38:16.947396 14355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.948733 14340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 04:38:16.948992 14340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "e8ed9aa998974ff78ec6023920c7b4de"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "f5ffe563d2fb4bc42accd55e9d6eb83a"
server_key_iv: "7b5f1365882065a91ed9a76617747f76"
server_key_version: "encryptionkey@0"
I20260620 04:38:16.949141 14340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:16.962316 14340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:16.965571 14340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:16.965811 14340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:16.966369 14340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:16.967154 14340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:16.967226 14340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.967289 14340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:16.967334 14340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:16.976581 14340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34919
I20260620 04:38:16.976600 14468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34919 every 8 connection(s)
I20260620 04:38:16.977437 14340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:16.985883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14340
I20260620 04:38:16.986019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:38:16.986312 392 external_mini_cluster.cc:1428] Setting key dfd5cf49f8d161ee00e6ff74b7449210
Jun 20 04:38:16 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:16.991727 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:16.979279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49815 (local address 127.0.98.62:42657)
0620 04:38:16.979436 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:16.979441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:16.980578 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 04:38:16.981531 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:16.981534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:16.981590 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:16.981666 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:16.982697 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.983214 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.984348 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.984567 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.984656 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:16.987372 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:16.987375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:16.987378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:16.987403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:16.989199 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.990025 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.990028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.990029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.990108 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:16.990561 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:16.990565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:16.990566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:16.990782 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:16.990874 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:16.991505 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 04:38:16.991611 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:38:16.992909 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:16.979684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:49815)
0620 04:38:16.980419 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:16.980458 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:16.981308 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:16.981788 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:16.981794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:16.982118 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:16.982554 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:16.982566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.983316 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.983318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:16.984156 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:16.984164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:16.984699 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:16.984701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:16.984833 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:16.985429 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:16.985444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:16.987221 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:16.989462 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:16.989467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:16.989477 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:16.989886 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:16.990223 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:16.990227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:16.990231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:16.990421 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:16.990904 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:16.990914 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:16.991226 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 04:38:16.991435 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 04:38:16.991777 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":128,"threads_started":1}
I20260620 04:38:16.994236 14469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:16.994508 14469 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:16.994979 14469 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:16.996057 14144 ts_manager.cc:194] Registered new tserver with Master: e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:38:16.996649 14144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:38:17.016028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:17.134111 14474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:17.134369 14474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:17.134411 14474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:17.138335 14474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:17.138393 14474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:17.138453 14474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:17.143859 14474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:17.144929 14474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:17.146144 14474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:17.155821 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:17.155853 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:17.155830 14482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:17.156082 14474 server_base.cc:1034] running on GCE node
I20260620 04:38:17.156675 14474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:17.157303 14474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:17.158476 14474 hybrid_clock.cc:648] HybridClock initialized: now 1781930297158459 us; error 29 us; skew 500 ppm
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:17.161369 14474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:17.162274 14474 webserver.cc:460] Webserver started at http://127.0.98.3:35043/ using document root <none> and password file <none>
I20260620 04:38:17.162772 14474 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:17.162829 14474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:17.162992 14474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:17.164973 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a23935fed2640eb9b0923cae3f688a5"
format_stamp: "Formatted at 2026-06-20 04:38:17 on dist-test-slave-zmf4"
server_key: "4fb39c504c8d580b1a00e4baaab9e93f"
server_key_iv: "8ea82553250fda3dc12ce0961c3b90cc"
server_key_version: "encryptionkey@0"
I20260620 04:38:17.165592 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a23935fed2640eb9b0923cae3f688a5"
format_stamp: "Formatted at 2026-06-20 04:38:17 on dist-test-slave-zmf4"
server_key: "4fb39c504c8d580b1a00e4baaab9e93f"
server_key_iv: "8ea82553250fda3dc12ce0961c3b90cc"
server_key_version: "encryptionkey@0"
I20260620 04:38:17.169389 14474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:38:17.171844 14489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:17.173169 14474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:17.173264 14474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "0a23935fed2640eb9b0923cae3f688a5"
format_stamp: "Formatted at 2026-06-20 04:38:17 on dist-test-slave-zmf4"
server_key: "4fb39c504c8d580b1a00e4baaab9e93f"
server_key_iv: "8ea82553250fda3dc12ce0961c3b90cc"
server_key_version: "encryptionkey@0"
I20260620 04:38:17.173491 14474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:17.185546 14474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:17.188802 14474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:17.189007 14474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:17.189589 14474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:17.190389 14474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:17.190440 14474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:17.190479 14474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:17.190495 14474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:17.199736 14474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33675
I20260620 04:38:17.199771 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33675 every 8 connection(s)
I20260620 04:38:17.200619 14474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:17.202819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14474
I20260620 04:38:17.202946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:38:17.203222 392 external_mini_cluster.cc:1428] Setting key 6599b67a66a77221302ace908093c315
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:17.214596 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.202489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34047 (local address 127.0.98.62:42657)
0620 04:38:17.202665 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.202671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.203826 (+ 1155us) server_negotiation.cc:407] Connection header received
0620 04:38:17.204817 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.204820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.204863 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.204928 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:17.206125 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.206661 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.207821 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.208043 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.208130 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.210653 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:17.210657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:17.210659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:17.210683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:17.212246 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.213011 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.213014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.213015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.213107 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.213500 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.213503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.213504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.213700 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:17.213800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.214368 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.214476 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:38:17.215618 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.202845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:34047)
0620 04:38:17.203665 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.203715 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.204590 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.205101 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.205109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.205479 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:17.205922 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.205934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.206802 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.206805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.207681 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.207690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.208173 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.208176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.208380 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.208898 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:17.208913 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:17.210463 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:17.212489 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.212494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.212506 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.212889 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.213231 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.213234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.213235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.213373 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.213823 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:17.213828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:17.214066 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.214304 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.214591 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":143,"threads_started":1}
I20260620 04:38:17.217089 14603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:17.217348 14603 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:17.217893 14603 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:17.218966 14143 ts_manager.cc:194] Registered new tserver with Master: 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:17.219547 14143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34047
I20260620 04:38:17.229622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:17.239311 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.231827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.0.98.62:42657)
0620 04:38:17.231967 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.231970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.232090 (+ 120us) server_negotiation.cc:407] Connection header received
0620 04:38:17.232145 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.232148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.232280 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.232370 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:17.233232 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.233752 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.234609 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.234817 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.234899 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.236282 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:17.236285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:17.236287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:17.236308 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:17.237713 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.238131 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.238135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.238155 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.238240 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.238532 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.238534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.238536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.238742 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:17.238846 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.239116 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.239202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:38:17.242758 14143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:38:17.245880 14143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:38:17.260839 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.254598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.0.1:34426)
0620 04:38:17.255489 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.255537 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.255667 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.256248 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.256254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.256276 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:17.256414 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.256420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.257698 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.257701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.259407 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.259414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260502 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.260613 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.260633 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.260696 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.260749 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":239,"thread_start_us":84,"threads_started":1}
I20260620 04:38:17.260838 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.255332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33675 (local address 127.0.0.1:42150)
0620 04:38:17.256502 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.256517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.256650 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.257286 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.257289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.257305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:17.257381 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.257386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.258430 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.258433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.259258 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.259270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260286 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.260392 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.260427 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.260547 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.260672 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260620 04:38:17.260963 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.255503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.0.98.3:33675)
0620 04:38:17.256277 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.256290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.256585 (+ 295us) server_negotiation.cc:407] Connection header received
0620 04:38:17.256809 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.256816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.256983 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.257098 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:17.257514 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.258293 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.259399 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260126 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260279 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.260425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.260712 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.260822 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":103,"threads_started":1}
I20260620 04:38:17.261624 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.254707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46275 (local address 127.0.0.1:59966)
0620 04:38:17.255894 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.255906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.255989 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.256558 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.256561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.256579 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:17.256672 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.256677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.258009 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.258012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.259229 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.259246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260490 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.261338 (+ 844us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.261361 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.261436 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.261504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":788,"thread_start_us":83,"threads_started":1}
I20260620 04:38:17.261627 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.255322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.0.98.2:34919)
0620 04:38:17.255679 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.255690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.255717 (+ 27us) server_negotiation.cc:407] Connection header received
0620 04:38:17.255800 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.255808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.256035 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.256577 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:17.256736 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.257569 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.259517 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260348 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.260623 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.261372 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.261489 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":148,"threads_started":1}
I20260620 04:38:17.261726 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.254881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.98.1:46275)
0620 04:38:17.255134 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.255150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.255982 (+ 832us) server_negotiation.cc:407] Connection header received
0620 04:38:17.256098 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.256103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.256414 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.256576 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:17.256784 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.257864 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.259533 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.260348 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.260591 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.260722 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.261479 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.261571 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260620 04:38:17.264338 14403 tablet_service.cc:1467] Processing CreateTablet for tablet fdd4df252d1445d1ae259af521fca7ec (DEFAULT_TABLE table=test-table [id=722d88fee12a4dcfa98b8b52c2c5290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:17.264362 14537 tablet_service.cc:1467] Processing CreateTablet for tablet fdd4df252d1445d1ae259af521fca7ec (DEFAULT_TABLE table=test-table [id=722d88fee12a4dcfa98b8b52c2c5290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:17.264362 14267 tablet_service.cc:1467] Processing CreateTablet for tablet fdd4df252d1445d1ae259af521fca7ec (DEFAULT_TABLE table=test-table [id=722d88fee12a4dcfa98b8b52c2c5290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:17.265308 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd4df252d1445d1ae259af521fca7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:17.265343 14267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd4df252d1445d1ae259af521fca7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:17.265349 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd4df252d1445d1ae259af521fca7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:17.272126 14621 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Bootstrap starting.
I20260620 04:38:17.272648 14622 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Bootstrap starting.
I20260620 04:38:17.272962 14620 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Bootstrap starting.
I20260620 04:38:17.274525 14621 tablet_bootstrap.cc:654] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:17.274525 14622 tablet_bootstrap.cc:654] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:17.274968 14620 tablet_bootstrap.cc:654] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:17.275686 14620 log.cc:824] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:17.275792 14621 log.cc:824] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:17.275799 14622 log.cc:824] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:17.277884 14620 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: No bootstrap required, opened a new log
I20260620 04:38:17.277984 14621 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: No bootstrap required, opened a new log
I20260620 04:38:17.278002 14622 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: No bootstrap required, opened a new log
I20260620 04:38:17.278179 14621 ts_tablet_manager.cc:1397] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:38:17.278179 14622 ts_tablet_manager.cc:1397] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:38:17.278213 14620 ts_tablet_manager.cc:1397] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:38:17.281101 14620 raft_consensus.cc:348] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.281442 14620 raft_consensus.cc:374] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:17.281499 14620 raft_consensus.cc:729] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ed9aa998974ff78ec6023920c7b4de, State: Initialized, Role: FOLLOWER
I20260620 04:38:17.281935 14620 consensus_queue.cc:260] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.281985 14621 raft_consensus.cc:348] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.281985 14622 raft_consensus.cc:348] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.282231 14621 raft_consensus.cc:374] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:17.282231 14622 raft_consensus.cc:374] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:17.282290 14622 raft_consensus.cc:729] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a23935fed2640eb9b0923cae3f688a5, State: Initialized, Role: FOLLOWER
I20260620 04:38:17.282298 14621 raft_consensus.cc:729] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Initialized, Role: FOLLOWER
I20260620 04:38:17.282713 14621 consensus_queue.cc:260] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.282714 14622 consensus_queue.cc:260] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.283126 14620 ts_tablet_manager.cc:1428] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:38:17.283488 14469 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
I20260620 04:38:17.283721 14621 ts_tablet_manager.cc:1428] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:38:17.283721 14622 ts_tablet_manager.cc:1428] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 04:38:17.284288 14335 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
I20260620 04:38:17.284313 14603 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
I20260620 04:38:17.428472 14627 raft_consensus.cc:482] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:17.428730 14627 raft_consensus.cc:504] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.429893 14627 leader_election.cc:288] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:17.434340 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.430263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.98.1:37947)
0620 04:38:17.430602 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.430615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.430764 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.431059 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.431062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.431088 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:17.431270 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.431279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.432216 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.432219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.433126 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.433136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.433895 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.433899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.433999 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.434028 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.434106 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.434174 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":20,"thread_start_us":117,"threads_started":1}
I20260620 04:38:17.434391 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.430376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37947 (local address 127.0.98.2:34919)
0620 04:38:17.430536 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.430541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.430707 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:38:17.430878 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.430882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.430942 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.431024 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:17.431475 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.432074 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.433264 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.433750 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.433831 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.433897 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.434166 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.434244 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:38:17.434612 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.430706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33675 (local address 127.0.98.1:56849)
0620 04:38:17.431040 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.431053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.431141 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.431477 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.431480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.431498 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:17.431607 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.431613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.432366 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.432369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.433361 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.433374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.434176 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.434180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.434284 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.434308 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.434392 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.434467 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260620 04:38:17.434643 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.430840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56849 (local address 127.0.98.3:33675)
0620 04:38:17.430976 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.430980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.431128 (+ 148us) server_negotiation.cc:407] Connection header received
0620 04:38:17.431229 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.431232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.431281 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.431368 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:17.431703 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.432238 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.433516 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.434008 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.434128 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.434183 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.434429 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.434513 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:38:17.435003 14423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd4df252d1445d1ae259af521fca7ec" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ed9aa998974ff78ec6023920c7b4de" is_pre_election: true
I20260620 04:38:17.435187 14557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd4df252d1445d1ae259af521fca7ec" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a23935fed2640eb9b0923cae3f688a5" is_pre_election: true
I20260620 04:38:17.435333 14423 raft_consensus.cc:2454] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 0.
I20260620 04:38:17.435420 14557 raft_consensus.cc:2454] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 0.
I20260620 04:38:17.435763 14225 leader_election.cc:302] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588, e8ed9aa998974ff78ec6023920c7b4de; no voters:
I20260620 04:38:17.436003 14627 raft_consensus.cc:2781] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:17.436089 14627 raft_consensus.cc:482] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:17.436133 14627 raft_consensus.cc:3037] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:17.437170 14627 raft_consensus.cc:504] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.437502 14627 leader_election.cc:288] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 election: Requested vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:17.437880 14423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd4df252d1445d1ae259af521fca7ec" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ed9aa998974ff78ec6023920c7b4de"
I20260620 04:38:17.437911 14557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd4df252d1445d1ae259af521fca7ec" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a23935fed2640eb9b0923cae3f688a5"
I20260620 04:38:17.437978 14423 raft_consensus.cc:3037] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:17.438051 14557 raft_consensus.cc:3037] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:17.439016 14423 raft_consensus.cc:2454] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 1.
I20260620 04:38:17.439091 14557 raft_consensus.cc:2454] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 1.
I20260620 04:38:17.439289 14225 leader_election.cc:302] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588, e8ed9aa998974ff78ec6023920c7b4de; no voters:
I20260620 04:38:17.439455 14627 raft_consensus.cc:2781] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:17.439669 14627 raft_consensus.cc:686] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 LEADER]: Becoming Leader. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Running, Role: LEADER
I20260620 04:38:17.439997 14627 consensus_queue.cc:237] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.442946 14140 catalog_manager.cc:5495] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2e11515aea16491baac8ab2ebe73e588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } health_report { overall_health: UNKNOWN } } }
W20260620 04:38:17.451897 14604 tablet.cc:2369] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:38:17.478722 14470 tablet.cc:2369] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:17.494870 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.491292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.98.1:46275)
0620 04:38:17.491447 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.491452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.491467 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:38:17.491572 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.491575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.491628 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.491737 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:17.492000 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.492567 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.493436 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.493656 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.493777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.494091 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:17.494465 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:17.494601 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.494659 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.494717 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:38:17.500835 14557 raft_consensus.cc:1261] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Refusing update from remote peer 2e11515aea16491baac8ab2ebe73e588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:17.500851 14423 raft_consensus.cc:1261] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Refusing update from remote peer 2e11515aea16491baac8ab2ebe73e588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:17.501586 14631 consensus_queue.cc:1035] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:17.501796 14627 consensus_queue.cc:1035] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:17.509845 14636 mvcc.cc:205] Tried to move back new op lower bound from 7298786498556190720 to 7298786498316820480. Current Snapshot: MvccSnapshot[applied={T|T < 7298786498556190720}]
I20260620 04:38:17.518587 14144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:17.518728 14144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:17.520980 14144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Sending DeleteTablet for 3 replicas of tablet fdd4df252d1445d1ae259af521fca7ec
I20260620 04:38:17.521744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14206
I20260620 04:38:17.521742 14267 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd4df252d1445d1ae259af521fca7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59966
I20260620 04:38:17.521773 14403 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd4df252d1445d1ae259af521fca7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34426
I20260620 04:38:17.521899 14537 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd4df252d1445d1ae259af521fca7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42150
I20260620 04:38:17.523229 14645 tablet_replica.cc:330] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: stopping tablet replica
I20260620 04:38:17.523579 14644 tablet_replica.cc:330] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: stopping tablet replica
I20260620 04:38:17.523561 14645 raft_consensus.cc:2229] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:17.523967 14644 raft_consensus.cc:2229] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:17.524022 14645 raft_consensus.cc:2258] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:17.524878 14644 raft_consensus.cc:2258] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:17.525661 14645 ts_tablet_manager.cc:1905] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:17.526248 14644 ts_tablet_manager.cc:1905] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:17.528510 14645 ts_tablet_manager.cc:1918] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:17.528606 14645 log.cc:1196] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/fdd4df252d1445d1ae259af521fca7ec
I20260620 04:38:17.528959 14645 ts_tablet_manager.cc:1939] T fdd4df252d1445d1ae259af521fca7ec P 0a23935fed2640eb9b0923cae3f688a5: Deleting consensus metadata
I20260620 04:38:17.529017 14644 ts_tablet_manager.cc:1918] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 04:38:17.529008 14129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46275 (error 108)
I20260620 04:38:17.529103 14644 log.cc:1196] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/fdd4df252d1445d1ae259af521fca7ec
I20260620 04:38:17.529395 14644 ts_tablet_manager.cc:1939] T fdd4df252d1445d1ae259af521fca7ec P e8ed9aa998974ff78ec6023920c7b4de: Deleting consensus metadata
I20260620 04:38:17.530077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14340
I20260620 04:38:17.530126 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.529836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46275 (local address 127.0.0.1:59988)
0620 04:38:17.529969 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.530039 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 04:38:17.530292 14131 catalog_manager.cc:4841] TS 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): tablet fdd4df252d1445d1ae259af521fca7ec (table test-table [id=722d88fee12a4dcfa98b8b52c2c5290c]) successfully deleted
W20260620 04:38:17.531510 14129 catalog_manager.cc:4584] TS 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1:46275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdd4df252d1445d1ae259af521fca7ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46275: connect: Connection refused (error 111)
I20260620 04:38:17.536209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14474
I20260620 04:38:17.537894 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.536419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.0.1:34432)
0620 04:38:17.537767 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.537815 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1277}
I20260620 04:38:17.542389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14112
I20260620 04:38:17.548624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--webserver_interface=127.0.98.62
--webserver_port=35975
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:17.657301 14647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:17.657588 14647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:17.657660 14647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:17.661576 14647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:17.661660 14647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:17.661700 14647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:17.661758 14647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:17.661794 14647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:17.666851 14647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:17.667845 14647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:17.668864 14647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:17.681090 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:17.681090 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:17.681262 14647 server_base.cc:1034] running on GCE node
W20260620 04:38:17.681152 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:17.682021 14647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:17.683162 14647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:17.684329 14647 hybrid_clock.cc:648] HybridClock initialized: now 1781930297684317 us; error 32 us; skew 500 ppm
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:17.687162 14647 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:17.688050 14647 webserver.cc:460] Webserver started at http://127.0.98.62:35975/ using document root <none> and password file <none>
I20260620 04:38:17.688668 14647 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:17.688771 14647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:17.692512 14647 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:38:17.694734 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:17.696125 14647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:17.696406 14647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "0df7d796ad894c448bc2e282258efe1c"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "87ee9d6a6203e05d6bba0fad1d6cebf9"
server_key_iv: "ba2ff1a7826ad92ab95d440eed9988e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:17.697057 14647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:17.726759 14647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:17.729812 14647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:17.730036 14647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:17.737485 14647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42657
I20260620 04:38:17.737766 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42657 every 8 connection(s)
I20260620 04:38:17.738603 14647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:17.742764 14716 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:38:17.744441 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap starting.
I20260620 04:38:17.746308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14647
I20260620 04:38:17.746740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46275
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=44509
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:17.761647 14716 log.cc:824] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:17.770084 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:17.770521 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap complete.
I20260620 04:38:17.774500 14716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:17.774909 14716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Initialized, Role: FOLLOWER
I20260620 04:38:17.775373 14716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:17.775519 14716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:17.775609 14716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:17.775730 14716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:38:17.777755 14716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:17.778080 14716 leader_election.cc:302] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df7d796ad894c448bc2e282258efe1c; no voters:
I20260620 04:38:17.778437 14716 leader_election.cc:288] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:38:17.778522 14720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:38:17.778713 14720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 LEADER]: Becoming Leader. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Running, Role: LEADER
I20260620 04:38:17.778999 14720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:17.779616 14716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:17.779847 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df7d796ad894c448bc2e282258efe1c. Latest consensus state: current_term: 2 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:17.779953 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:17.780318 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:17.780405 14721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:17.780589 14721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:17.785275 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:17.786026 14724 catalog_manager.cc:1259] Loaded cluster ID: 43680da639494e369a96a6a1e6e5be3a
I20260620 04:38:17.786104 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:17.787197 14724 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 04:38:17.789971 14724 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 04:38:17.794111 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:17.794716 14724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Loaded TSK: 0
I20260620 04:38:17.795624 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:38:17.870045 14718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:17.870344 14718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:17.870422 14718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:17.874914 14718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:17.875017 14718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:17.875110 14718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:17.881029 14718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44509
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:17.882104 14718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:17.883195 14718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:17.892948 14743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:17.892952 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:17.892976 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:17.893492 14718 server_base.cc:1034] running on GCE node
I20260620 04:38:17.893954 14718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:17.894824 14718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:17.896050 14718 hybrid_clock.cc:648] HybridClock initialized: now 1781930297896029 us; error 36 us; skew 500 ppm
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:17.899665 14718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:17.900804 14718 webserver.cc:460] Webserver started at http://127.0.98.1:44509/ using document root <none> and password file <none>
I20260620 04:38:17.901523 14718 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:17.901641 14718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:17.906839 14718 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:38:17.909560 14753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:17.911072 14718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:38:17.911208 14718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "2e11515aea16491baac8ab2ebe73e588"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "4c14643ee073f91a076180f3620936b5"
server_key_iv: "62b8ad234a3e1e56bb5f4428f363c6af"
server_key_version: "encryptionkey@0"
I20260620 04:38:17.911923 14718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:17.938059 14718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:17.942006 14718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:17.942193 14718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:17.942835 14718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:17.944017 14760 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:38:17.948006 14718 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:38:17.948077 14718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:38:17.948136 14718 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:38:17.950053 14718 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:38:17.950105 14718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:38:17.950469 14760 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Bootstrap starting.
I20260620 04:38:17.960747 14718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46275
I20260620 04:38:17.960860 14867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46275 every 8 connection(s)
I20260620 04:38:17.961678 14718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:17.963775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14718
I20260620 04:38:17.964399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=39983
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:17.970885 14760 log.cc:824] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:17.978497 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.963772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:39299)
0620 04:38:17.964517 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.964567 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.965320 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.966678 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.966692 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.967245 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:17.967868 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.967887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.969287 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.969297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.970231 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.970241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.970967 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.970971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.971166 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.971880 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:17.971909 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:17.973378 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:17.975446 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.975452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.975464 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.975717 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.976084 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.976087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.976089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.976283 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.976873 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:17.976879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:17.977113 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.977287 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.977620 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":149,"threads_started":1}
I20260620 04:38:17.978483 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.963752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39299 (local address 127.0.98.62:42657)
0620 04:38:17.964239 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.964258 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.965549 (+ 1291us) server_negotiation.cc:407] Connection header received
0620 04:38:17.965787 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.965804 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.966197 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.966549 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:17.968071 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.969114 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.970362 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.970810 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.971000 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.973572 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:17.973577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:17.973589 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:17.973615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:17.975299 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.975836 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.975841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.975848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.975974 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.976383 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.976388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.976391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.976724 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:17.976900 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.977410 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.977662 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":119,"threads_started":1}
I20260620 04:38:17.980711 14760 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:17.981004 14868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:17.981169 14760 tablet_bootstrap.cc:492] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Bootstrap complete.
I20260620 04:38:17.981266 14868 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:17.981787 14868 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:17.982597 14760 ts_tablet_manager.cc:1397] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260620 04:38:17.984282 14680 ts_manager.cc:194] Registered new tserver with Master: 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1:46275)
I20260620 04:38:17.985570 14760 raft_consensus.cc:348] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.986142 14760 raft_consensus.cc:729] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Initialized, Role: FOLLOWER
I20260620 04:38:17.986667 14760 consensus_queue.cc:260] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } }
I20260620 04:38:17.987625 14760 ts_tablet_manager.cc:1428] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:38:17.988953 14680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39299
I20260620 04:38:17.991498 14868 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
Jun 20 04:38:17 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:38:17.997877 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.987162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46275 (local address 127.0.0.1:59992)
0620 04:38:17.987537 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:17.987578 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:17.987747 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:17.988578 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:17.988584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:17.988935 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:17.989031 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.989061 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.990149 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.990152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.991043 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:17.991054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.991782 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.991801 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:17.991911 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.992522 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:17.992533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:17.994025 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:17.995968 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.995978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.995983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.996266 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.996697 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:17.996702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:17.996704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:17.996803 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:17.997243 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:17.997249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:17.997371 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 04:38:17.997510 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 04:38:17.997698 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":134,"threads_started":1}
I20260620 04:38:17.997888 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:17.987899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.1:46275)
0620 04:38:17.988153 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:17.988164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:17.988185 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:38:17.988291 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:17.988302 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:17.988440 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:17.988577 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:17.989195 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.990021 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.991188 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:17.991662 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:17.992215 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:17.994173 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:17.994181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:17.994193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:17.994222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:17.995842 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.996377 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.996383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.996386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.996466 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:17.996904 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:17.996908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:17.996910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:17.997130 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:17.997269 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:17.997598 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 04:38:17.997749 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
I20260620 04:38:17.998764 14802 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd4df252d1445d1ae259af521fca7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59992
I20260620 04:38:17.999526 14877 tablet_replica.cc:330] stopping tablet replica
I20260620 04:38:17.999796 14877 raft_consensus.cc:2229] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:18.000028 14877 raft_consensus.cc:2258] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:18.000980 14877 ts_tablet_manager.cc:1905] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:18.004359 14877 ts_tablet_manager.cc:1918] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:18.004439 14877 log.cc:1196] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/fdd4df252d1445d1ae259af521fca7ec
I20260620 04:38:18.004733 14877 ts_tablet_manager.cc:1939] T fdd4df252d1445d1ae259af521fca7ec P 2e11515aea16491baac8ab2ebe73e588: Deleting consensus metadata
I20260620 04:38:18.006250 14665 catalog_manager.cc:4841] TS 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1:46275): tablet fdd4df252d1445d1ae259af521fca7ec (table test-table [id=722d88fee12a4dcfa98b8b52c2c5290c]) successfully deleted
W20260620 04:38:18.085187 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:18.085421 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:18.085462 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:18.089371 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:18.089424 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:18.089480 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:18.094671 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39983
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:18.095650 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:18.096674 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:18.106472 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.106453 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.106454 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:18.106570 14872 server_base.cc:1034] running on GCE node
I20260620 04:38:18.107165 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:18.107937 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:18.109135 14872 hybrid_clock.cc:648] HybridClock initialized: now 1781930298109137 us; error 50 us; skew 500 ppm
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:18.111964 14872 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:18.112877 14872 webserver.cc:460] Webserver started at http://127.0.98.2:39983/ using document root <none> and password file <none>
I20260620 04:38:18.113438 14872 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:18.113548 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:18.117446 14872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:38:18.119602 14892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.120935 14872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:38:18.121033 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "e8ed9aa998974ff78ec6023920c7b4de"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "f5ffe563d2fb4bc42accd55e9d6eb83a"
server_key_iv: "7b5f1365882065a91ed9a76617747f76"
server_key_version: "encryptionkey@0"
I20260620 04:38:18.121666 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:18.157970 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:18.161252 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:18.161412 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:18.161949 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:18.162715 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:18.162765 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.162803 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:18.162818 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.172879 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34919
I20260620 04:38:18.172933 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34919 every 8 connection(s)
I20260620 04:38:18.173934 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:18.183426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 14872
I20260620 04:38:18.184105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33675
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=35043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:18.190349 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.176181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35727 (local address 127.0.98.62:42657)
0620 04:38:18.176362 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.176365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.177368 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 04:38:18.178256 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.178259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.178336 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.178442 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:18.179739 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.180448 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.181610 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.181864 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.181978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.185866 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:18.185872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:18.185877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:18.185916 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:18.187827 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.188537 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.188541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.188543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.188632 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.189093 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.189096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.189097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.189308 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:18.189390 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.190101 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.190215 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 04:38:18.191605 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.176485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:35727)
0620 04:38:18.177131 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.177169 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.178054 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.178617 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.178623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.178978 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:18.179553 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.179568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.180557 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.180560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.181482 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.181492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.182016 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.182020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.182188 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.182889 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:18.182940 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:18.185653 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:18.187969 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:18.187978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:18.187996 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:18.188424 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:18.188737 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:18.188742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:18.188747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:18.188949 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:18.189423 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:18.189440 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:18.189793 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.190025 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.190418 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":113,"threads_started":1}
I20260620 04:38:18.193030 15006 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:18.193279 15006 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:18.193763 15006 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:18.194826 14680 ts_manager.cc:194] Registered new tserver with Master: e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919)
I20260620 04:38:18.195729 14680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35727
W20260620 04:38:18.298128 15009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:18.298379 15009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:18.298422 15009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:18.302459 15009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:18.302515 15009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:18.302573 15009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:18.307737 15009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35043
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:18.308849 15009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:18.309865 15009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:18.319352 15014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.319351 15017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.319367 15015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:18.319667 15009 server_base.cc:1034] running on GCE node
I20260620 04:38:18.320271 15009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:18.320868 15009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:18.322028 15009 hybrid_clock.cc:648] HybridClock initialized: now 1781930298321996 us; error 46 us; skew 500 ppm
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:18.324860 15009 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:18.325766 15009 webserver.cc:460] Webserver started at http://127.0.98.3:35043/ using document root <none> and password file <none>
I20260620 04:38:18.326314 15009 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:18.326388 15009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:18.330282 15009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:18.332475 15024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.333732 15009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:38:18.333837 15009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "0a23935fed2640eb9b0923cae3f688a5"
format_stamp: "Formatted at 2026-06-20 04:38:17 on dist-test-slave-zmf4"
server_key: "4fb39c504c8d580b1a00e4baaab9e93f"
server_key_iv: "8ea82553250fda3dc12ce0961c3b90cc"
server_key_version: "encryptionkey@0"
I20260620 04:38:18.334408 15009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:18.364519 15009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:18.368111 15009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:18.368343 15009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:18.368952 15009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:18.369781 15009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:18.369850 15009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.369932 15009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:18.369984 15009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.379308 15009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33675
I20260620 04:38:18.379370 15137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33675 every 8 connection(s)
I20260620 04:38:18.380441 15009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:18.381729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15009
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:18.392011 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.382368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.0.98.62:42657)
0620 04:38:18.382616 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.382622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.382788 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:38:18.382956 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.382958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.383002 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.383090 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:18.383959 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.384788 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.386154 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.386413 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.386489 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.388037 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:18.388044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:18.388050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:18.388082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:18.389899 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.390432 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.390435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.390436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.390510 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.390872 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.390878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.390882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.391062 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:18.391225 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.391471 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.391765 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260620 04:38:18.395144 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.382363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45863 (local address 127.0.98.62:42657)
0620 04:38:18.382490 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.382496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.383806 (+ 1310us) server_negotiation.cc:407] Connection header received
0620 04:38:18.384964 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.384968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.385020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.385118 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:18.386304 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.386977 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.388126 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.388434 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.388545 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.391445 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:18.391449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:18.391451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:18.391482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:18.392941 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.393645 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.393649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.393650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.393738 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:18.394107 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:18.394111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:18.394113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:18.394325 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:18.394428 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.394945 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.395038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":1408}
I20260620 04:38:18.395975 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.382859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:45863)
0620 04:38:18.383632 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.383706 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.384749 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.385232 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.385237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.385577 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:18.386021 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.386034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.387103 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.387107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.388007 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.388016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.388613 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.388634 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.388856 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.389388 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:18.389410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:18.391240 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:18.393068 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:18.393073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:18.393092 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:18.393467 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:18.393859 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:18.393862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:18.393863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:18.394010 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:18.394474 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:18.394482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:18.394720 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.394878 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.395130 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":153,"threads_started":1}
I20260620 04:38:18.397372 15138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42657
I20260620 04:38:18.397596 15138 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:18.398025 15138 heartbeater.cc:511] Master 127.0.98.62:42657 requested a full tablet report, sending...
I20260620 04:38:18.398988 14680 ts_manager.cc:194] Registered new tserver with Master: 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:18.399559 14680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45863
I20260620 04:38:18.406244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:18.410849 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.406894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.0.98.62:42657)
0620 04:38:18.407045 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.407049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.407222 (+ 173us) server_negotiation.cc:407] Connection header received
0620 04:38:18.407375 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.407378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.407484 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.407582 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:18.407801 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.408480 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.409386 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.409603 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.409691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.410066 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:18.410457 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:18.410549 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.410704 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.410752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 04:38:18.411446 14680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:38:18.414777 14680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:38:18.422760 14802 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 (DEFAULT_TABLE table=test-table [id=b5c66b0f479e405e9f68d790112427f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:18.423408 14802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:18.426580 14760 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Bootstrap starting.
I20260620 04:38:18.427377 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.421957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33675 (local address 127.0.0.1:42156)
0620 04:38:18.422262 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.422276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.422358 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.423235 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.423238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.423256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:18.423346 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.423353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.424709 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.424712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.425849 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.425859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.426942 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.426946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.427043 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.427091 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.427144 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.427221 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":41,"thread_start_us":116,"threads_started":1}
I20260620 04:38:18.427474 14760 tablet_bootstrap.cc:654] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:18.427799 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.421652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.0.1:34444)
0620 04:38:18.421780 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.421797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.421880 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.422786 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.422788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.422806 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:18.422915 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.422923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.424357 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.424360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.425838 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.425849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.427427 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.427430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.427523 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.427542 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.427608 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.427683 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":88}
I20260620 04:38:18.427881 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.421862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.0.98.2:34919)
0620 04:38:18.422205 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.422233 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.422257 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:38:18.422348 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.422354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.422580 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.422801 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:18.423041 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.424228 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.425980 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.427279 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.427433 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.427585 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.427641 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.427736 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260620 04:38:18.428136 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.422413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.0.98.3:33675)
0620 04:38:18.422742 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.422754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.422784 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:38:18.422858 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.422862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.423091 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.423238 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:18.423473 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.424554 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.425979 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.426794 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.427663 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.427816 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.427892 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.427979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
I20260620 04:38:18.429011 14760 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: No bootstrap required, opened a new log
I20260620 04:38:18.429096 14760 ts_tablet_manager.cc:1397] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:18.429632 14760 raft_consensus.cc:348] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.429766 14760 raft_consensus.cc:374] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:18.429805 14760 raft_consensus.cc:729] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Initialized, Role: FOLLOWER
I20260620 04:38:18.429965 14760 consensus_queue.cc:260] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.430305 14760 ts_tablet_manager.cc:1428] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:38:18.431182 15072 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 (DEFAULT_TABLE table=test-table [id=b5c66b0f479e405e9f68d790112427f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:18.431181 14940 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 (DEFAULT_TABLE table=test-table [id=b5c66b0f479e405e9f68d790112427f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:18.432271 15072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:18.432276 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:18.439030 15151 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Bootstrap starting.
I20260620 04:38:18.439050 15152 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Bootstrap starting.
I20260620 04:38:18.441803 15152 tablet_bootstrap.cc:654] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:18.441810 15151 tablet_bootstrap.cc:654] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:18.442549 15152 log.cc:824] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:18.442550 15151 log.cc:824] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:18.444357 15152 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: No bootstrap required, opened a new log
I20260620 04:38:18.444370 15151 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: No bootstrap required, opened a new log
I20260620 04:38:18.444554 15152 ts_tablet_manager.cc:1397] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:38:18.444561 15151 ts_tablet_manager.cc:1397] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:38:18.447661 15151 raft_consensus.cc:348] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.447894 15151 raft_consensus.cc:374] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:18.447696 15152 raft_consensus.cc:348] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.447952 15152 raft_consensus.cc:374] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:18.447937 15151 raft_consensus.cc:729] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a23935fed2640eb9b0923cae3f688a5, State: Initialized, Role: FOLLOWER
I20260620 04:38:18.448035 15152 raft_consensus.cc:729] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ed9aa998974ff78ec6023920c7b4de, State: Initialized, Role: FOLLOWER
I20260620 04:38:18.448480 15152 consensus_queue.cc:260] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.448482 15151 consensus_queue.cc:260] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.449144 15006 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
I20260620 04:38:18.449462 15151 ts_tablet_manager.cc:1428] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:38:18.449544 15152 ts_tablet_manager.cc:1428] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:38:18.450290 15138 heartbeater.cc:503] Master 127.0.98.62:42657 was elected leader, sending a full tablet report...
W20260620 04:38:18.462841 14869 tablet.cc:2369] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:18.500488 14874 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:18.500773 14874 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.501968 14874 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:18.506337 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.502196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.98.1:57855)
0620 04:38:18.502531 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.502547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.502706 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.503089 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.503092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.503111 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:18.503225 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.503230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.504118 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.504122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.505100 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.505109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.505830 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.505834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.505951 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.506009 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.506084 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.506160 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
I20260620 04:38:18.506448 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.502714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46017 (local address 127.0.98.3:33675)
0620 04:38:18.502869 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.502875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.503055 (+ 180us) server_negotiation.cc:407] Connection header received
0620 04:38:18.503429 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.503432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.503475 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.503534 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:18.503794 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.504414 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.505414 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.505886 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.505997 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.506057 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.506248 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.506321 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:38:18.506450 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.502355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57855 (local address 127.0.98.2:34919)
0620 04:38:18.502527 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.502532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.502636 (+ 104us) server_negotiation.cc:407] Connection header received
0620 04:38:18.502915 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.502919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.502980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.503060 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:18.503336 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.504011 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.505219 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.505688 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.505766 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.505818 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.506237 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.506317 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 04:38:18.506649 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.502602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33675 (local address 127.0.98.1:46017)
0620 04:38:18.502945 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.502959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.503267 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:18.503570 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:18.503573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:18.503590 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:18.503696 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.503701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.504511 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.504514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.505271 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:18.505281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.506003 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.506006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:18.506118 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.506140 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:38:18.506218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:38:18.506286 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":66,"threads_started":1}
I20260620 04:38:18.507138 14960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec8dcd78dd640dfa06f7dfd797e56f3" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ed9aa998974ff78ec6023920c7b4de" is_pre_election: true
I20260620 04:38:18.507432 14960 raft_consensus.cc:2454] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 0.
I20260620 04:38:18.507917 14757 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588, e8ed9aa998974ff78ec6023920c7b4de; no voters:
I20260620 04:38:18.508237 14874 raft_consensus.cc:2781] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:18.508327 14874 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:18.508257 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec8dcd78dd640dfa06f7dfd797e56f3" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a23935fed2640eb9b0923cae3f688a5" is_pre_election: true
I20260620 04:38:18.508383 14874 raft_consensus.cc:3037] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:18.508543 15092 raft_consensus.cc:2454] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 0.
I20260620 04:38:18.509748 14874 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.510141 14874 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 election: Requested vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
I20260620 04:38:18.510571 14960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec8dcd78dd640dfa06f7dfd797e56f3" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ed9aa998974ff78ec6023920c7b4de"
I20260620 04:38:18.510547 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec8dcd78dd640dfa06f7dfd797e56f3" candidate_uuid: "2e11515aea16491baac8ab2ebe73e588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a23935fed2640eb9b0923cae3f688a5"
I20260620 04:38:18.510727 14960 raft_consensus.cc:3037] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:18.510752 15092 raft_consensus.cc:3037] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:18.511729 14960 raft_consensus.cc:2454] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 1.
I20260620 04:38:18.511751 15092 raft_consensus.cc:2454] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e11515aea16491baac8ab2ebe73e588 in term 1.
I20260620 04:38:18.512003 14757 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a23935fed2640eb9b0923cae3f688a5, 2e11515aea16491baac8ab2ebe73e588; no voters:
I20260620 04:38:18.512180 14874 raft_consensus.cc:2781] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:18.512392 14874 raft_consensus.cc:686] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 LEADER]: Becoming Leader. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Running, Role: LEADER
I20260620 04:38:18.512660 14874 consensus_queue.cc:237] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:18.515723 14680 catalog_manager.cc:5495] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2e11515aea16491baac8ab2ebe73e588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } health_report { overall_health: HEALTHY } } }
I20260620 04:38:18.528164 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.524203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.98.1:46275)
0620 04:38:18.524548 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:18.524555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:18.524566 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:38:18.524627 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:18.524630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:18.524677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:18.524772 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:18.524994 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.525772 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.526737 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:18.526956 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:18.527036 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:18.527284 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:18.527707 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:18.527799 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:18.527921 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 04:38:18.527968 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260620 04:38:18.532138 15092 raft_consensus.cc:1261] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Refusing update from remote peer 2e11515aea16491baac8ab2ebe73e588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:18.532572 14960 raft_consensus.cc:1261] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Refusing update from remote peer 2e11515aea16491baac8ab2ebe73e588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:18.533018 15159 consensus_queue.cc:1035] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:18.533408 14874 consensus_queue.cc:1035] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:18.544047 15164 mvcc.cc:205] Tried to move back new op lower bound from 7298786502781108224 to 7298786502710792192. Current Snapshot: MvccSnapshot[applied={T|T < 7298786502781108224}]
I20260620 04:38:18.548276 14680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:18.548471 14680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:18.550688 14680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Sending DeleteTablet for 3 replicas of tablet 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:18.551393 15072 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42156
I20260620 04:38:18.551512 14802 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59992
I20260620 04:38:18.551546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14718
I20260620 04:38:18.551530 14940 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34444
I20260620 04:38:18.552415 15174 tablet_replica.cc:330] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: stopping tablet replica
I20260620 04:38:18.552724 15174 raft_consensus.cc:2229] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:18.553010 15174 raft_consensus.cc:2258] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:18.554327 15174 ts_tablet_manager.cc:1905] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:18.556504 15174 ts_tablet_manager.cc:1918] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:18.556633 15174 log.cc:1196] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:18.556854 15173 tablet_replica.cc:330] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: stopping tablet replica
I20260620 04:38:18.557024 15174 ts_tablet_manager.cc:1939] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P e8ed9aa998974ff78ec6023920c7b4de: Deleting consensus metadata
I20260620 04:38:18.557191 15173 raft_consensus.cc:2229] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:18.557525 15173 raft_consensus.cc:2258] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:18.558254 14667 catalog_manager.cc:4841] TS e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 (table test-table [id=b5c66b0f479e405e9f68d790112427f1]) successfully deleted
W20260620 04:38:18.558620 14665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46275 (error 108)
I20260620 04:38:18.559142 15173 ts_tablet_manager.cc:1905] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:18.559543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14872
I20260620 04:38:18.559567 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:18.559295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46275 (local address 127.0.0.1:60000)
0620 04:38:18.559431 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.559492 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 04:38:18.560178 14665 catalog_manager.cc:4584] TS 2e11515aea16491baac8ab2ebe73e588 (127.0.98.1:46275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ec8dcd78dd640dfa06f7dfd797e56f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46275: connect: Connection refused (error 111)
I20260620 04:38:18.562428 15173 ts_tablet_manager.cc:1918] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:18.562543 15173 log.cc:1196] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:18.562891 15173 ts_tablet_manager.cc:1939] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 0a23935fed2640eb9b0923cae3f688a5: Deleting consensus metadata
I20260620 04:38:18.564167 14667 catalog_manager.cc:4841] TS 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): tablet 8ec8dcd78dd640dfa06f7dfd797e56f3 (table test-table [id=b5c66b0f479e405e9f68d790112427f1]) successfully deleted
I20260620 04:38:18.567602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15009
I20260620 04:38:18.573223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 14647
I20260620 04:38:18.579631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--webserver_interface=127.0.98.62
--webserver_port=35975
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:18.689801 15176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:18.690083 15176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:18.690152 15176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:18.694303 15176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:18.694403 15176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:18.694444 15176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:18.694505 15176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:18.694545 15176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:18.699766 15176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:18.700790 15176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:18.701789 15176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:18.710146 15184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.710158 15181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:18.710208 15176 server_base.cc:1034] running on GCE node
W20260620 04:38:18.710158 15182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:18.710886 15176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:18.711957 15176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:18.713145 15176 hybrid_clock.cc:648] HybridClock initialized: now 1781930298713101 us; error 46 us; skew 500 ppm
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:18.715888 15176 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:18.716818 15176 webserver.cc:460] Webserver started at http://127.0.98.62:35975/ using document root <none> and password file <none>
I20260620 04:38:18.717358 15176 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:18.717422 15176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:18.721098 15176 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:18.723254 15191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.724537 15176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:18.724671 15176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "0df7d796ad894c448bc2e282258efe1c"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "87ee9d6a6203e05d6bba0fad1d6cebf9"
server_key_iv: "ba2ff1a7826ad92ab95d440eed9988e2"
server_key_version: "encryptionkey@0"
I20260620 04:38:18.725204 15176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:18.754408 15176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:18.757611 15176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:18.757843 15176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:18.765524 15176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42657
I20260620 04:38:18.765763 15243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42657 every 8 connection(s)
I20260620 04:38:18.766685 15176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:18.771346 15244 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:38:18.773289 15244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap starting.
I20260620 04:38:18.776181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15176
I20260620 04:38:18.776677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46275
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=44509
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:18.790910 15244 log.cc:824] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:18.801824 15244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:18.802196 15244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c: Bootstrap complete.
I20260620 04:38:18.806149 15244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:18.806540 15244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Initialized, Role: FOLLOWER
I20260620 04:38:18.807060 15244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:18.807204 15244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:18.807291 15244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:18.807400 15244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:38:18.809288 15244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:18.809612 15244 leader_election.cc:302] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df7d796ad894c448bc2e282258efe1c; no voters:
I20260620 04:38:18.809947 15244 leader_election.cc:288] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:38:18.810027 15248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:38:18.810209 15248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [term 3 LEADER]: Becoming Leader. State: Replica: 0df7d796ad894c448bc2e282258efe1c, State: Running, Role: LEADER
I20260620 04:38:18.810494 15248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } }
I20260620 04:38:18.811124 15244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:18.811340 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:18.811455 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:18.811656 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df7d796ad894c448bc2e282258efe1c. Latest consensus state: current_term: 3 leader_uuid: "0df7d796ad894c448bc2e282258efe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df7d796ad894c448bc2e282258efe1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42657 } } }
I20260620 04:38:18.811738 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df7d796ad894c448bc2e282258efe1c [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:18.812011 15252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:18.817072 15252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:18.817739 15252 catalog_manager.cc:1259] Loaded cluster ID: 43680da639494e369a96a6a1e6e5be3a
I20260620 04:38:18.817817 15252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 04:38:18.818619 15252 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbd97dc49ad google::LogMessage::Fail() at ??:0
@ 0x7fbd97dc788a google::LogMessage::SendToLog() at ??:0
@ 0x7fbd97dc44de google::LogMessage::Flush() at ??:0
@ 0x7fbd97dc5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbda2703fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbda2704d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbda2701579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbda2745640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbd9a0163d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbd9a016c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbd9a0183f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbd9a007512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbd9b64b6db start_thread at ??:0
@ 0x7fbd968c261f clone at ??:0
W20260620 04:38:18.902585 15246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:18.902863 15246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:18.902963 15246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:18.907229 15246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:18.907320 15246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:18.907428 15246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:18.912808 15246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44509
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:18.914039 15246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:18.915105 15246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:18.925524 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.925525 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:18.925525 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:18.926223 15246 server_base.cc:1034] running on GCE node
I20260620 04:38:18.926615 15246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:18.927243 15246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:18.928423 15246 hybrid_clock.cc:648] HybridClock initialized: now 1781930298928409 us; error 31 us; skew 500 ppm
Jun 20 04:38:18 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:18.931313 15246 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:18.932142 15246 webserver.cc:460] Webserver started at http://127.0.98.1:44509/ using document root <none> and password file <none>
I20260620 04:38:18.932724 15246 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:18.932832 15246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:18.936856 15246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:18.939011 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:18.940404 15246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:18.940542 15246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "2e11515aea16491baac8ab2ebe73e588"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "4c14643ee073f91a076180f3620936b5"
server_key_iv: "62b8ad234a3e1e56bb5f4428f363c6af"
server_key_version: "encryptionkey@0"
I20260620 04:38:18.941082 15246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:18.960556 15246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:18.963739 15246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:18.963961 15246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:18.964709 15246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:18.965826 15302 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:38:18.971467 15246 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:38:18.971565 15246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 04:38:18.971653 15246 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:38:18.973357 15246 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:38:18.973429 15246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:38:18.973704 15302 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Bootstrap starting.
I20260620 04:38:18.984097 15246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46275
I20260620 04:38:18.984136 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46275 every 8 connection(s)
I20260620 04:38:18.985234 15246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:18.993314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15246
I20260620 04:38:18.993877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=39983
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:18.997179 15302 log.cc:824] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:19.016688 15302 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:19.017120 15302 tablet_bootstrap.cc:492] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Bootstrap complete.
I20260620 04:38:19.018585 15302 ts_tablet_manager.cc:1397] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.004s
I20260620 04:38:19.021757 15302 raft_consensus.cc:348] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:19.022284 15302 raft_consensus.cc:729] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e11515aea16491baac8ab2ebe73e588, State: Initialized, Role: FOLLOWER
I20260620 04:38:19.022787 15302 consensus_queue.cc:260] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:19.023838 15302 ts_tablet_manager.cc:1428] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260620 04:38:19.117383 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:38:18.987227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:57435)
0620 04:38:18.987707 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:18.987759 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:18.988638 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:19.116604 (+127966us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":146,"threads_started":1}
W20260620 04:38:19.117913 15410 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42657 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42657: Client connection negotiation failed: client connection to 127.0.98.62:42657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:38:19.118650 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.118394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:33241)
0620 04:38:19.118516 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.118593 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 04:38:19.119268 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.119064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:57495)
0620 04:38:19.119165 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.119212 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 04:38:19.119436 15410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:38:19.152487 15413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:19.152758 15413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:19.152841 15413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:19.157316 15413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:19.157403 15413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:19.157500 15413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:19.163237 15413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39983
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:19.164345 15413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:19.165488 15413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:38:19.175278 15413 server_base.cc:1034] running on GCE node
W20260620 04:38:19.175312 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:19.175436 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:19.175289 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:19.175990 15413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:19.176636 15413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:19.177803 15413 hybrid_clock.cc:648] HybridClock initialized: now 1781930299177786 us; error 34 us; skew 500 ppm
Jun 20 04:38:19 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:19.180564 15413 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:19.181339 15413 webserver.cc:460] Webserver started at http://127.0.98.2:39983/ using document root <none> and password file <none>
I20260620 04:38:19.181841 15413 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:19.181903 15413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:19.185812 15413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:19.188094 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.189510 15413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:19.189613 15413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "e8ed9aa998974ff78ec6023920c7b4de"
format_stamp: "Formatted at 2026-06-20 04:38:16 on dist-test-slave-zmf4"
server_key: "f5ffe563d2fb4bc42accd55e9d6eb83a"
server_key_iv: "7b5f1365882065a91ed9a76617747f76"
server_key_version: "encryptionkey@0"
I20260620 04:38:19.190191 15413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:19.205832 15413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:19.209079 15413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:19.209232 15413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:19.209828 15413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:19.210635 15413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:19.210707 15413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.210775 15413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:19.210827 15413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.221104 15413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34919
I20260620 04:38:19.221124 15543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34919 every 8 connection(s)
I20260620 04:38:19.222056 15413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:19.224416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15413
I20260620 04:38:19.224905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33675
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42657
--webserver_port=35043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:19.226054 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.224122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:39811)
0620 04:38:19.225004 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.225161 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":129,"threads_started":1}
W20260620 04:38:19.226688 15544 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42657 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42657: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
I20260620 04:38:19.227335 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.227101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:53753)
0620 04:38:19.227233 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.227282 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:38:19.228075 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.227848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:38777)
0620 04:38:19.227972 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.228026 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 04:38:19.228338 15544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:38:19.336377 15547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:19.336804 15547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:19.336933 15547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:19.340938 15547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:19.341028 15547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:19.341121 15547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:19.346503 15547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35043
--tserver_master_addrs=127.0.98.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:19.347559 15547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:19.348590 15547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:19.357911 15555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:19.357971 15553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:19.358165 15552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:19.357951 15547 server_base.cc:1034] running on GCE node
I20260620 04:38:19.358641 15547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:19.359297 15547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:19.360520 15547 hybrid_clock.cc:648] HybridClock initialized: now 1781930299360474 us; error 62 us; skew 500 ppm
Jun 20 04:38:19 dist-test-slave-zmf4 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:19.363307 15547 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:19.364261 15547 webserver.cc:460] Webserver started at http://127.0.98.3:35043/ using document root <none> and password file <none>
I20260620 04:38:19.364768 15547 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:19.364827 15547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:19.368651 15547 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:38:19.370802 15562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.372136 15547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:38:19.372309 15547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "0a23935fed2640eb9b0923cae3f688a5"
format_stamp: "Formatted at 2026-06-20 04:38:17 on dist-test-slave-zmf4"
server_key: "4fb39c504c8d580b1a00e4baaab9e93f"
server_key_iv: "8ea82553250fda3dc12ce0961c3b90cc"
server_key_version: "encryptionkey@0"
I20260620 04:38:19.372834 15547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:19.390113 15547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:19.393352 15547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:19.393568 15547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:19.394250 15547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:19.395117 15547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:19.395164 15547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.395231 15547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:19.395267 15547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:19.404239 15547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33675
I20260620 04:38:19.404297 15675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33675 every 8 connection(s)
I20260620 04:38:19.405310 15547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:19.409093 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.407383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:40101)
0620 04:38:19.408328 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.408444 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":524,"threads_started":1}
W20260620 04:38:19.409603 15676 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42657 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42657: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
I20260620 04:38:19.410197 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.409967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:45653)
0620 04:38:19.410088 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.410145 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 04:38:19.411012 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:19.410795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:38241)
0620 04:38:19.410914 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:19.410957 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:38:19.411249 15676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:38:19.412380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15547
I20260620 04:38:20.120939 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.120361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:47621)
0620 04:38:20.120749 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:20.120826 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":87,"threads_started":1}
I20260620 04:38:20.229372 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.228947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:32835)
0620 04:38:20.229212 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:20.229269 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260620 04:38:20.412321 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.411797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:59557)
0620 04:38:20.412081 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:20.412136 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260620 04:38:20.875976 15683 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:20.876160 15683 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:20.877396 15683 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
Jun 20 04:38:20 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 04:38:20 dist-test-slave-zmf4 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:38:20.892901 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.877945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33675 (local address 127.0.98.1:59069)
0620 04:38:20.878196 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:20.878210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:20.878304 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:20.879862 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:20.879868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:20.880466 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:20.880972 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:20.880988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.882258 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.882265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:20.883214 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:20.883223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.883938 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.883943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:20.884176 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:20.884951 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:20.884964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:20.886843 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:20.889665 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:20.889671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:20.889685 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:20.890221 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:20.890678 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:20.890684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:20.890686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:20.890872 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:20.891751 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:20.891758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:20.892026 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 04:38:20.892294 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 04:38:20.892673 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":106,"spinlock_wait_cycles":5120,"thread_start_us":63,"threads_started":1}
I20260620 04:38:20.892926 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.878094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59069 (local address 127.0.98.3:33675)
0620 04:38:20.878361 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:20.878378 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:20.878409 (+ 31us) server_negotiation.cc:407] Connection header received
0620 04:38:20.879035 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:20.879067 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:20.879457 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:20.879775 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:20.881150 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.882114 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.883366 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.883817 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.883991 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:20.887127 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:20.887132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:20.887143 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:20.887170 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:20.889514 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:20.890360 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:20.890368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:20.890377 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:20.890529 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:20.891008 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:20.891011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:20.891012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:20.891625 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:20.891725 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:20.892424 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 04:38:20.892739 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260620 04:38:20.892920 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.877613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34919 (local address 127.0.98.1:34391)
0620 04:38:20.877863 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:20.877881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:20.878000 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:20.880175 (+ 2175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:20.880178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:20.880461 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:20.880973 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:20.880990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.882317 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.882320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:20.883215 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:20.883224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.883966 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.883970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:20.884177 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:20.884908 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:20.884924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:20.887353 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:20.890366 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:20.890369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:20.890371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:20.890603 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:20.891274 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:20.891277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:20.891280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:20.891369 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:20.892156 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:20.892158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:20.892299 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 04:38:20.892410 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 04:38:20.892759 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":327,"spinlock_wait_cycles":2944,"thread_start_us":66,"threads_started":1}
I20260620 04:38:20.893466 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:20.877762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34391 (local address 127.0.98.2:34919)
0620 04:38:20.878292 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:20.878310 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:20.878358 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:38:20.879242 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:20.879249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:20.879663 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:20.880241 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:20.881146 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.882162 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.883343 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:20.883820 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:20.884003 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:20.887578 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:20.887585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:20.887600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:20.887638 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:20.890251 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:20.890712 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:20.890719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:20.890727 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:20.890965 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:20.891496 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:20.891499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:20.891501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:20.892032 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:20.892185 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:20.892559 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 04:38:20.892833 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":81,"threads_started":1}
W20260620 04:38:20.895566 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
W20260620 04:38:20.895805 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:20.895890 15299 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588; no voters: 0a23935fed2640eb9b0923cae3f688a5, e8ed9aa998974ff78ec6023920c7b4de
I20260620 04:38:20.896248 15683 raft_consensus.cc:2726] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:38:21.122027 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:21.121728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:40879)
0620 04:38:21.121890 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:21.121943 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 04:38:21.230767 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:21.230202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:39319)
0620 04:38:21.230602 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:21.230655 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":119,"threads_started":1}
I20260620 04:38:21.413584 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:21.413142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:50181)
0620 04:38:21.413433 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:21.413495 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260620 04:38:22.123349 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:22.122905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:60085)
0620 04:38:22.123199 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:22.123256 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260620 04:38:22.231912 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:22.231537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:45675)
0620 04:38:22.231765 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:22.231818 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260620 04:38:22.415007 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:22.414531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:48415)
0620 04:38:22.414840 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:22.414902 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260620 04:38:23.082756 15693 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:23.082928 15693 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:23.083353 15693 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
W20260620 04:38:23.084292 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
W20260620 04:38:23.084431 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:23.084491 15299 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588; no voters: 0a23935fed2640eb9b0923cae3f688a5, e8ed9aa998974ff78ec6023920c7b4de
I20260620 04:38:23.084666 15693 raft_consensus.cc:2726] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:38:23.124682 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:23.124141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:49117)
0620 04:38:23.124498 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:23.124567 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":160,"threads_started":1}
I20260620 04:38:23.233196 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:23.232731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:51121)
0620 04:38:23.233017 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:23.233074 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260620 04:38:23.416311 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:23.415817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:46973)
0620 04:38:23.416104 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:23.416165 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260620 04:38:24.126308 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:24.125623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:37069)
0620 04:38:24.126037 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:24.126181 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":80,"threads_started":1}
I20260620 04:38:24.234503 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:24.234030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:55217)
0620 04:38:24.234335 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:24.234398 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260620 04:38:24.417575 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:24.417146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:33579)
0620 04:38:24.417421 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:24.417480 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260620 04:38:25.128144 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:25.127480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:38199)
0620 04:38:25.127831 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:25.128042 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":95,"threads_started":1}
I20260620 04:38:25.235853 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:25.235384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:44823)
0620 04:38:25.235679 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:25.235739 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260620 04:38:25.418802 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:25.418370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:57787)
0620 04:38:25.418638 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:25.418693 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260620 04:38:26.129492 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:26.129061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:37811)
0620 04:38:26.129340 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:26.129397 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260620 04:38:26.237169 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:26.236677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:40061)
0620 04:38:26.237005 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:26.237074 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260620 04:38:26.412333 15705 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:26.412485 15705 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:26.412869 15705 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
W20260620 04:38:26.413757 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
W20260620 04:38:26.414083 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:26.414140 15299 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588; no voters: 0a23935fed2640eb9b0923cae3f688a5, e8ed9aa998974ff78ec6023920c7b4de
I20260620 04:38:26.414274 15705 raft_consensus.cc:2726] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:38:26.420055 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:26.419632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:38373)
0620 04:38:26.419919 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:26.419969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260620 04:38:27.131227 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:27.130708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:48867)
0620 04:38:27.131015 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:27.131100 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":127,"threads_started":1}
I20260620 04:38:27.238683 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:27.238199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:44783)
0620 04:38:27.238524 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:27.238587 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
I20260620 04:38:27.421413 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:27.420943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:48721)
0620 04:38:27.421246 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:27.421307 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260620 04:38:28.132550 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:28.132082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:54415)
0620 04:38:28.132401 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:28.132457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":137,"threads_started":1}
I20260620 04:38:28.239930 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:28.239518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:39575)
0620 04:38:28.239784 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:28.239836 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260620 04:38:28.422672 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:28.422246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:44209)
0620 04:38:28.422519 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:28.422578 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260620 04:38:29.114732 15713 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:29.114876 15713 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:29.115260 15713 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
W20260620 04:38:29.116124 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
W20260620 04:38:29.116442 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:29.116511 15299 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588; no voters: 0a23935fed2640eb9b0923cae3f688a5, e8ed9aa998974ff78ec6023920c7b4de
I20260620 04:38:29.116657 15713 raft_consensus.cc:2726] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:38:29.134011 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:29.133603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:35499)
0620 04:38:29.133865 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:29.133920 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260620 04:38:29.241176 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:29.240739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:39989)
0620 04:38:29.241025 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:29.241079 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260620 04:38:29.423939 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:29.423498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:57391)
0620 04:38:29.423781 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:29.423837 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260620 04:38:30.135426 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:30.134958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:33941)
0620 04:38:30.135236 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:30.135302 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260620 04:38:30.242465 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:30.242003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:59579)
0620 04:38:30.242310 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:30.242367 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":74,"threads_started":1}
I20260620 04:38:30.425724 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:30.425155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:52553)
0620 04:38:30.425479 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:30.425610 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":92,"threads_started":1}
I20260620 04:38:31.136790 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:31.136341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:38265)
0620 04:38:31.136623 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:31.136680 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260620 04:38:31.243712 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:31.243265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:47527)
0620 04:38:31.243561 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:31.243615 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":74,"threads_started":1}
I20260620 04:38:31.427095 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:31.426658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:60241)
0620 04:38:31.426955 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:31.427005 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":133,"threads_started":1}
I20260620 04:38:32.137980 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:32.137573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:46009)
0620 04:38:32.137826 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:32.137880 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260620 04:38:32.245097 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:32.244597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:41577)
0620 04:38:32.244885 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:32.244983 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20260620 04:38:32.428344 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:32.427886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:59595)
0620 04:38:32.428163 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:32.428245 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260620 04:38:33.139137 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:33.138740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:34435)
0620 04:38:33.138995 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:33.139044 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260620 04:38:33.246680 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:33.246017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:60181)
0620 04:38:33.246495 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:33.246563 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260620 04:38:33.429612 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:33.429156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.3:54583)
0620 04:38:33.429455 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:33.429515 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260620 04:38:33.915485 15729 raft_consensus.cc:482] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:33.915627 15729 raft_consensus.cc:504] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ed9aa998974ff78ec6023920c7b4de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34919 } } peers { permanent_uuid: "0a23935fed2640eb9b0923cae3f688a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33675 } } peers { permanent_uuid: "2e11515aea16491baac8ab2ebe73e588" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46275 } }
I20260620 04:38:33.916146 15729 leader_election.cc:288] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919), 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675)
W20260620 04:38:33.917002 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8ed9aa998974ff78ec6023920c7b4de (127.0.98.2:34919): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
W20260620 04:38:33.917115 15299 leader_election.cc:341] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a23935fed2640eb9b0923cae3f688a5 (127.0.98.3:33675): Not found: Tablet not found: 8ec8dcd78dd640dfa06f7dfd797e56f3
I20260620 04:38:33.917157 15299 leader_election.cc:302] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e11515aea16491baac8ab2ebe73e588; no voters: 0a23935fed2640eb9b0923cae3f688a5, e8ed9aa998974ff78ec6023920c7b4de
I20260620 04:38:33.917325 15729 raft_consensus.cc:2726] T 8ec8dcd78dd640dfa06f7dfd797e56f3 P 2e11515aea16491baac8ab2ebe73e588 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:38:34.140583 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:34.140139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.1:51133)
0620 04:38:34.140428 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:34.140484 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":129,"threads_started":1}
I20260620 04:38:34.248090 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:34.247670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42657 (local address 127.0.98.2:51139)
0620 04:38:34.247928 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:34.247991 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260620 04:38:34.416291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15246
I20260620 04:38:34.422441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15413
I20260620 04:38:34.428038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15547
2026-06-20T04:38:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19332 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:38:34 dist-test-slave-zmf4 krb5kdc[15734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:38:34 dist-test-slave-zmf4 krb5kdc[15734](info): set up 2 sockets
Jun 20 04:38:34 dist-test-slave-zmf4 krb5kdc[15734](info): commencing operation
krb5kdc: starting...
W20260620 04:38:35.532441 392 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:38:35 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:38:35.655368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:35.769421 15747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:35.769752 15747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:35.769829 15747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:35.774201 15747 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:38:35.774470 15747 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:38:35.774555 15747 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:38:35.774592 15747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:35.774693 15747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:35.774792 15747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:35.774860 15747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:35.774923 15747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:35.780797 15747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:35.781915 15747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:35.782946 15747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:35.791687 15755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:35.791697 15752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:35.791767 15747 server_base.cc:1034] running on GCE node
W20260620 04:38:35.791698 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:35.792572 15747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:35.793725 15747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:35.795048 15747 hybrid_clock.cc:648] HybridClock initialized: now 1781930315795014 us; error 55 us; skew 500 ppm
Jun 20 04:38:35 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:35.798101 15747 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:35.798951 15747 webserver.cc:460] Webserver started at http://127.0.98.62:37957/ using document root <none> and password file <none>
I20260620 04:38:35.799515 15747 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:35.799607 15747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:35.799834 15747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:35.801962 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e08c7c5c89431ca8b9963d75af45c1"
format_stamp: "Formatted at 2026-06-20 04:38:35 on dist-test-slave-zmf4"
server_key: "c78bf204b3520d00a6a164bcfd43d20c"
server_key_iv: "be11caf3f757a32082b1257a6d80bb4f"
server_key_version: "encryptionkey@0"
I20260620 04:38:35.802516 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e08c7c5c89431ca8b9963d75af45c1"
format_stamp: "Formatted at 2026-06-20 04:38:35 on dist-test-slave-zmf4"
server_key: "c78bf204b3520d00a6a164bcfd43d20c"
server_key_iv: "be11caf3f757a32082b1257a6d80bb4f"
server_key_version: "encryptionkey@0"
I20260620 04:38:35.806402 15747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:38:35.808859 15762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:35.810132 15747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:35.810285 15747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "b0e08c7c5c89431ca8b9963d75af45c1"
format_stamp: "Formatted at 2026-06-20 04:38:35 on dist-test-slave-zmf4"
server_key: "c78bf204b3520d00a6a164bcfd43d20c"
server_key_iv: "be11caf3f757a32082b1257a6d80bb4f"
server_key_version: "encryptionkey@0"
I20260620 04:38:35.810402 15747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:35.829201 15747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:35.832716 15747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:35.832962 15747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:35.840401 15747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40751
I20260620 04:38:35.840719 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40751 every 8 connection(s)
I20260620 04:38:35.841383 15747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:35.841881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15747
I20260620 04:38:35.842012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:38:35.842284 392 external_mini_cluster.cc:1428] Setting key eda1d82e9978272a8c8b4e96d769f826
I20260620 04:38:35.844691 15815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:35.851351 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap starting.
Jun 20 04:38:35 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:35.854172 15815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:35.855052 15815 log.cc:824] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:35.857523 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: No bootstrap required, opened a new log
I20260620 04:38:35.857822 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:35.843710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.62:40751)
0620 04:38:35.844112 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:35.844127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:35.844160 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:38:35.844997 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:35.845016 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:35.845330 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:35.845606 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:35.846503 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:35.847525 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:35.848669 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:35.849137 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:35.849374 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:35.852395 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:35.852401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:35.852417 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:35.852449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:35.854562 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:35.855027 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:35.855032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:35.855038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:35.855197 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:35.855703 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:35.855706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:35.855708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:35.856292 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:35.856444 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:35.856763 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 04:38:35.857073 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
I20260620 04:38:35.860498 15815 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:35.860730 15815 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:35.860812 15815 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Initialized, Role: FOLLOWER
I20260620 04:38:35.861210 15815 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:35.861357 15815 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:35.861443 15815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:35.861559 15815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:35.862440 15815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:35.862764 15815 leader_election.cc:302] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e08c7c5c89431ca8b9963d75af45c1; no voters:
I20260620 04:38:35.863072 15815 leader_election.cc:288] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:38:35.863307 15820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:35.863565 15820 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 LEADER]: Becoming Leader. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Running, Role: LEADER
I20260620 04:38:35.863864 15820 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:35.864373 15815 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:35.865595 15822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e08c7c5c89431ca8b9963d75af45c1. Latest consensus state: current_term: 1 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:38:35.865731 15822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:35.866119 15829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:35.866094 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:38:35.866394 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:35.868888 15829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:35.874577 15829 catalog_manager.cc:1347] Generated new cluster ID: ca1fe912743a4e518a3492adb41650c7
I20260620 04:38:35.874653 15829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:35.885046 15829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:38:35.885989 15829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:35.892498 15829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Generated new TSK 0
I20260620 04:38:35.893239 15829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:38:35.927066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:36.060650 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:36.060909 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:36.060961 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:36.064992 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:36.065053 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:36.065140 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:36.070447 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:36.071416 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:36.072436 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:36.082291 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:36.082296 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:36.082376 15841 server_base.cc:1034] running on GCE node
W20260620 04:38:36.082322 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:36.083004 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:36.083567 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:36.084800 15841 hybrid_clock.cc:648] HybridClock initialized: now 1781930316084782 us; error 31 us; skew 500 ppm
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:36.087859 15841 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:36.088797 15841 webserver.cc:460] Webserver started at http://127.0.98.1:35885/ using document root <none> and password file <none>
I20260620 04:38:36.089358 15841 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:36.089447 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:36.089679 15841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:36.091679 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "d05ed3faa13edb5509871afdd9cd4e80"
server_key_iv: "7c27945e5c0ada0b9fb84dc140538469"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.092315 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "d05ed3faa13edb5509871afdd9cd4e80"
server_key_iv: "7c27945e5c0ada0b9fb84dc140538469"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.096047 15841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:36.098722 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.100042 15841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:38:36.100236 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "d05ed3faa13edb5509871afdd9cd4e80"
server_key_iv: "7c27945e5c0ada0b9fb84dc140538469"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.100369 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:36.112035 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:36.115504 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:36.115706 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:36.116348 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:36.117187 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:36.117261 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.117344 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:36.117394 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.126400 15841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37433
I20260620 04:38:36.126529 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37433 every 8 connection(s)
I20260620 04:38:36.127477 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:36.134718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15841
I20260620 04:38:36.134869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:38:36.135154 392 external_mini_cluster.cc:1428] Setting key fa74f9d08b14f17f23ad30d7f3e764aa
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:36.142701 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.129437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46961 (local address 127.0.98.62:40751)
0620 04:38:36.129621 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.129626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.130424 (+ 798us) server_negotiation.cc:407] Connection header received
0620 04:38:36.131665 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.131668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.131715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.131790 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:36.133029 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.133567 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.134915 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.135189 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.135325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.138457 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:36.138464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:36.138471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:36.138509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:36.140301 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.140966 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.140968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.140970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.141046 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.141475 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.141478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.141479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.141664 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:36.141753 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.142449 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.142574 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 04:38:36.143939 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.129801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:46961)
0620 04:38:36.130264 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.130306 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.131461 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.131955 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.131962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.132401 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:36.132867 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.132881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.133702 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.133705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.134758 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.134770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.135380 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.135384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.135578 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.136516 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:36.136538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:36.138270 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:36.140434 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.140443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.140463 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.140851 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.141143 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.141149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.141154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.141351 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.141780 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:36.141787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:36.142145 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.142391 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.142798 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260620 04:38:36.145442 15970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:38:36.145936 15970 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:36.147073 15970 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:38:36.149008 15779 ts_manager.cc:194] Registered new tserver with Master: 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:36.150588 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:38:36.165874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:36.280016 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:36.280361 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:36.280457 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:36.284443 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:36.284530 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:36.284622 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:36.290128 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:36.291198 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:36.292230 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:36.302089 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:36.302090 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:36.302415 15975 server_base.cc:1034] running on GCE node
W20260620 04:38:36.302114 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:36.302845 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:36.303485 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:36.304670 15975 hybrid_clock.cc:648] HybridClock initialized: now 1781930316304638 us; error 51 us; skew 500 ppm
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:36.308068 15975 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:36.309638 15975 webserver.cc:460] Webserver started at http://127.0.98.2:40825/ using document root <none> and password file <none>
I20260620 04:38:36.310230 15975 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:36.310345 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:36.310570 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:36.312669 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "552b276207fc46d19e2887968912b780"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a06f6836ebe6a373f54cf6dc9b10dc50"
server_key_iv: "a51bd4599d2a77e0f87699dba87b5abe"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.313450 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "552b276207fc46d19e2887968912b780"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a06f6836ebe6a373f54cf6dc9b10dc50"
server_key_iv: "a51bd4599d2a77e0f87699dba87b5abe"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.317437 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 04:38:36.320029 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.321745 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:36.321878 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "552b276207fc46d19e2887968912b780"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a06f6836ebe6a373f54cf6dc9b10dc50"
server_key_iv: "a51bd4599d2a77e0f87699dba87b5abe"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.321992 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:36.348395 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:36.351630 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:36.351787 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:36.352396 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:36.353183 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:36.353235 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.353276 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:36.353291 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.362910 15975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43215
I20260620 04:38:36.362946 16103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43215 every 8 connection(s)
I20260620 04:38:36.363807 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:36.372835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 15975
I20260620 04:38:36.372974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:38:36.373402 392 external_mini_cluster.cc:1428] Setting key 8a45421cc1cc8959df66dcf6b13af67a
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:36.377768 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.365712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47021 (local address 127.0.98.62:40751)
0620 04:38:36.365844 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.365848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.367368 (+ 1520us) server_negotiation.cc:407] Connection header received
0620 04:38:36.368335 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.368338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.368396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.368474 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:36.369599 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.370130 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.371239 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.371468 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.371580 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.374018 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:36.374021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:36.374024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:36.374048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:36.375542 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.376051 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.376056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.376058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.376144 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.376530 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.376534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.376536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.376853 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:36.376992 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.377475 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.377599 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:38:36.378556 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.366118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:47021)
0620 04:38:36.367222 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.367263 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.368075 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.368638 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.368645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.368995 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:36.369458 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.369471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.370257 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.370262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.371114 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.371123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.371618 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.371622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.371789 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.372414 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:36.372430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:36.373838 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:36.375677 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.375683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.375702 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.375946 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.376280 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.376285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.376287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.376430 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.376987 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:36.376993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:36.377229 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.377420 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.377735 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":635,"threads_started":1}
I20260620 04:38:36.379757 16104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
I20260620 04:38:36.380033 16104 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:36.380692 16104 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
I20260620 04:38:36.381963 15779 ts_manager.cc:194] Registered new tserver with Master: 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:38:36.382808 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:38:36.401517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:36.517527 16109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:36.517763 16109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:36.517807 16109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:36.521636 16109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:36.521690 16109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:36.521747 16109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:36.528126 16109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:36.529371 16109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:36.530754 16109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:36.540729 16114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:36.540729 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:36.540737 16115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:36.540933 16109 server_base.cc:1034] running on GCE node
I20260620 04:38:36.541461 16109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:36.542068 16109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:36.543285 16109 hybrid_clock.cc:648] HybridClock initialized: now 1781930316543273 us; error 29 us; skew 500 ppm
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:36.546121 16109 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:36.546893 16109 webserver.cc:460] Webserver started at http://127.0.98.3:39163/ using document root <none> and password file <none>
I20260620 04:38:36.547389 16109 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:36.547446 16109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:36.547595 16109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:38:36.549526 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "3da1e206bb31410dba2353577e4ed362"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a07276f4910e6535cce60dec27b64e11"
server_key_iv: "e79616b2e5b69ff181f5f7ef0282e073"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.550025 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3da1e206bb31410dba2353577e4ed362"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a07276f4910e6535cce60dec27b64e11"
server_key_iv: "e79616b2e5b69ff181f5f7ef0282e073"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.553766 16109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:38:36.556036 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.557397 16109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:36.557498 16109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "3da1e206bb31410dba2353577e4ed362"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a07276f4910e6535cce60dec27b64e11"
server_key_iv: "e79616b2e5b69ff181f5f7ef0282e073"
server_key_version: "encryptionkey@0"
I20260620 04:38:36.557658 16109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:36.578486 16109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:36.581903 16109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:36.582119 16109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:36.582696 16109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:36.583477 16109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:36.583524 16109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.583590 16109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:36.583633 16109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:36.593114 16109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38457
I20260620 04:38:36.593142 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38457 every 8 connection(s)
I20260620 04:38:36.593953 16109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:38:36.598256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16109
I20260620 04:38:36.598379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:38:36.598682 392 external_mini_cluster.cc:1428] Setting key 8a585cdebb244f1fe6cc27c60d9c643b
Jun 20 04:38:36 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:36.608358 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.595722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36161 (local address 127.0.98.62:40751)
0620 04:38:36.595892 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.595897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.596682 (+ 785us) server_negotiation.cc:407] Connection header received
0620 04:38:36.598048 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.598054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.598116 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.598221 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:36.599646 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.600507 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.602053 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.602281 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.602366 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.604934 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:36.604944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:36.604946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:36.604970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:36.606384 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.606904 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.606907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.606908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.606987 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.607284 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.607286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.607287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.607450 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:36.607525 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.608052 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.608165 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:38:36.609114 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.596060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:36161)
0620 04:38:36.596536 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.596579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.597827 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.598432 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.598442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.598899 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:36.599506 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.599521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.600646 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.600651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.601919 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.601930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.602425 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.602428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.602590 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.603129 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:36.603145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:36.604768 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:36.606508 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.606513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.606526 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.606815 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.607069 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:36.607071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:36.607073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:36.607205 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:36.607573 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:36.607578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:36.607812 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.607972 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.608307 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":162,"threads_started":1}
I20260620 04:38:36.610491 16238 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
I20260620 04:38:36.610699 16238 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:36.611119 16238 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
I20260620 04:38:36.612059 15779 ts_manager.cc:194] Registered new tserver with Master: 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457)
I20260620 04:38:36.612830 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36161
I20260620 04:38:36.612947 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:36.622593 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.614949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.0.98.62:40751)
0620 04:38:36.615067 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.615070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.615249 (+ 179us) server_negotiation.cc:407] Connection header received
0620 04:38:36.615335 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.615338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.615370 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.615433 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:36.616019 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.616552 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.617412 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.617624 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.617715 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.619185 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:36.619188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:36.619190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:36.619210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:36.620792 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.621294 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.621297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.621299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.621373 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:36.621691 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:36.621694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:36.621695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:36.621915 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:36.622035 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.622348 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.622456 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:38:36.625423 15779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:38:36.628624 15779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:38:36.643489 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.638404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.0.98.2:43215)
0620 04:38:36.638749 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.638761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.638786 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:38:36.638869 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.638874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.639081 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.639327 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:36.639611 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.640560 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.642131 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.642807 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.642935 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.643088 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.643239 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.643328 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260620 04:38:36.643478 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.637538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.0.1:42874)
0620 04:38:36.637868 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.637917 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.638050 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.639312 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.639318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.639342 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:36.639491 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.639498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.640686 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.640691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.642002 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.642014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.642945 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.642949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.643053 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.643089 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.643203 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.643325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"spinlock_wait_cycles":10240,"thread_start_us":108,"threads_started":1}
I20260620 04:38:36.643752 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.638270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.0.1:50652)
0620 04:38:36.638975 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.638988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.639065 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.639721 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.639724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.639742 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:36.639840 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.639846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.641174 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.641177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.642356 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.642366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.643351 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.643355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.643450 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.643481 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.643530 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.643603 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":24,"thread_start_us":93,"threads_started":1}
I20260620 04:38:36.644369 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.637612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38457 (local address 127.0.0.1:48954)
0620 04:38:36.637867 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.637916 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.638049 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.639496 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.639499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.639516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:36.639618 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.639627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.641121 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.641124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.642265 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.642278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.643497 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.643501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.644042 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.644063 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.644142 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.644245 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":50,"spinlock_wait_cycles":6272,"thread_start_us":97,"threads_started":1}
I20260620 04:38:36.644443 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.638702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.98.1:37433)
0620 04:38:36.639294 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.639311 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.639331 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:38:36.639402 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.639406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.639582 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.639731 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:36.639959 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.641061 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.642461 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.643193 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.644001 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.644139 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.644186 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.644304 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":127,"threads_started":1}
I20260620 04:38:36.644734 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.638362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.98.3:38457)
0620 04:38:36.638828 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.638839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.638862 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:38:36.638952 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.638957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.639156 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.639312 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:36.639750 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.640975 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.642383 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.643193 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.643307 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.643443 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.644219 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.644318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":166,"threads_started":1}
I20260620 04:38:36.646453 16038 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d (DEFAULT_TABLE table=test-table [id=e8bc13b68fb14055a47fbf11a442232d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:36.647416 16038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0c18e99a5a4571b8359186f1f4244d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:36.649525 15904 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d (DEFAULT_TABLE table=test-table [id=e8bc13b68fb14055a47fbf11a442232d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:36.650233 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d (DEFAULT_TABLE table=test-table [id=e8bc13b68fb14055a47fbf11a442232d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:36.650552 15904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0c18e99a5a4571b8359186f1f4244d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:36.651147 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0c18e99a5a4571b8359186f1f4244d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:36.655090 16255 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Bootstrap starting.
I20260620 04:38:36.657168 16256 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Bootstrap starting.
I20260620 04:38:36.657683 16257 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap starting.
I20260620 04:38:36.658495 16255 tablet_bootstrap.cc:654] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:36.659031 16256 tablet_bootstrap.cc:654] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:36.659582 16255 log.cc:824] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:36.659766 16257 tablet_bootstrap.cc:654] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:36.660406 16256 log.cc:824] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:36.660710 16257 log.cc:824] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:36.662184 16255 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: No bootstrap required, opened a new log
I20260620 04:38:36.662284 16256 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: No bootstrap required, opened a new log
I20260620 04:38:36.662395 16255 ts_tablet_manager.cc:1397] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 04:38:36.662424 16256 ts_tablet_manager.cc:1397] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:38:36.662621 16257 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: No bootstrap required, opened a new log
I20260620 04:38:36.662807 16257 ts_tablet_manager.cc:1397] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:38:36.666424 16257 raft_consensus.cc:348] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.666702 16257 raft_consensus.cc:374] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:36.666754 16257 raft_consensus.cc:729] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7b4fad258c45f2ba85adfb061f66b5, State: Initialized, Role: FOLLOWER
I20260620 04:38:36.666580 16256 raft_consensus.cc:348] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.666580 16255 raft_consensus.cc:348] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.666891 16255 raft_consensus.cc:374] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:36.666891 16256 raft_consensus.cc:374] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:36.666953 16255 raft_consensus.cc:729] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552b276207fc46d19e2887968912b780, State: Initialized, Role: FOLLOWER
I20260620 04:38:36.666953 16256 raft_consensus.cc:729] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da1e206bb31410dba2353577e4ed362, State: Initialized, Role: FOLLOWER
I20260620 04:38:36.667260 16257 consensus_queue.cc:260] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.667397 16255 consensus_queue.cc:260] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.667397 16256 consensus_queue.cc:260] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.668064 16104 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
I20260620 04:38:36.668337 16257 ts_tablet_manager.cc:1428] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 04:38:36.668349 16256 ts_tablet_manager.cc:1428] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 04:38:36.668349 16255 ts_tablet_manager.cc:1428] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20260620 04:38:36.668649 15970 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
I20260620 04:38:36.669246 16238 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
W20260620 04:38:36.844940 16239 tablet.cc:2369] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:38:36.864995 16105 tablet.cc:2369] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:38:36.878993 15971 tablet.cc:2369] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:36.913308 16262 raft_consensus.cc:482] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:36.913559 16262 raft_consensus.cc:504] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.914959 16262 leader_election.cc:288] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 552b276207fc46d19e2887968912b780 (127.0.98.2:43215), 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:36.919991 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.915643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39233 (local address 127.0.98.1:37433)
0620 04:38:36.915764 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.915769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.915921 (+ 152us) server_negotiation.cc:407] Connection header received
0620 04:38:36.916018 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.916020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.916088 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.916161 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:36.916459 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.917142 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.918567 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.919270 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.919395 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.919465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.919780 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.919859 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:38:36.919991 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.915312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54143 (local address 127.0.98.2:43215)
0620 04:38:36.915430 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:36.915434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:36.915470 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:38:36.915577 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:36.915581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:36.915634 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:36.915730 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:36.916107 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.916930 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.918567 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.919269 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.919395 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.919465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:36.919780 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 04:38:36.919870 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:38:36.920029 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.915200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.98.3:54143)
0620 04:38:36.915338 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.915357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.915471 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.915816 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.915819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.915840 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:36.915985 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.915994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.917072 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.917075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.918402 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.918413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.919423 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.919427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.919594 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.919637 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.919722 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.919866 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":41}
I20260620 04:38:36.920037 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:36.915504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.98.3:39233)
0620 04:38:36.915829 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:36.915842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:36.915927 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:36.916180 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:36.916182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:36.916240 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:36.916344 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.916348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.917264 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.917267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.918410 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:36.918416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:36.919422 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:36.919427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:36.919622 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:36.919639 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:38:36.919722 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:38:36.919866 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":114,"thread_start_us":135,"threads_started":1}
I20260620 04:38:36.920794 15924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0c18e99a5a4571b8359186f1f4244d" candidate_uuid: "3da1e206bb31410dba2353577e4ed362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" is_pre_election: true
I20260620 04:38:36.920794 16058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0c18e99a5a4571b8359186f1f4244d" candidate_uuid: "3da1e206bb31410dba2353577e4ed362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552b276207fc46d19e2887968912b780" is_pre_election: true
I20260620 04:38:36.921113 16058 raft_consensus.cc:2454] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da1e206bb31410dba2353577e4ed362 in term 0.
I20260620 04:38:36.921113 15924 raft_consensus.cc:2454] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da1e206bb31410dba2353577e4ed362 in term 0.
I20260620 04:38:36.921588 16125 leader_election.cc:302] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780; no voters:
I20260620 04:38:36.921836 16262 raft_consensus.cc:2781] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:36.921924 16262 raft_consensus.cc:482] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:36.921967 16262 raft_consensus.cc:3037] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:36.922967 16262 raft_consensus.cc:504] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.923301 16262 leader_election.cc:288] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [CANDIDATE]: Term 1 election: Requested vote from peers 552b276207fc46d19e2887968912b780 (127.0.98.2:43215), 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:36.923703 15924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0c18e99a5a4571b8359186f1f4244d" candidate_uuid: "3da1e206bb31410dba2353577e4ed362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
I20260620 04:38:36.923703 16058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0c18e99a5a4571b8359186f1f4244d" candidate_uuid: "3da1e206bb31410dba2353577e4ed362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552b276207fc46d19e2887968912b780"
I20260620 04:38:36.923825 15924 raft_consensus.cc:3037] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:36.923839 16058 raft_consensus.cc:3037] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:36.925118 15924 raft_consensus.cc:2454] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da1e206bb31410dba2353577e4ed362 in term 1.
I20260620 04:38:36.925119 16058 raft_consensus.cc:2454] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da1e206bb31410dba2353577e4ed362 in term 1.
I20260620 04:38:36.925520 16125 leader_election.cc:302] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da1e206bb31410dba2353577e4ed362, 3f7b4fad258c45f2ba85adfb061f66b5; no voters:
I20260620 04:38:36.925679 16262 raft_consensus.cc:2781] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:36.925896 16262 raft_consensus.cc:686] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 1 LEADER]: Becoming Leader. State: Replica: 3da1e206bb31410dba2353577e4ed362, State: Running, Role: LEADER
I20260620 04:38:36.926211 16262 consensus_queue.cc:237] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:36.929114 15779 catalog_manager.cc:5495] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da1e206bb31410dba2353577e4ed362 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3da1e206bb31410dba2353577e4ed362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } health_report { overall_health: UNKNOWN } } }
I20260620 04:38:37.007782 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:37.003871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.98.3:38457)
0620 04:38:37.003996 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:37.004003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:37.004040 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:38:37.004258 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:37.004261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:37.004317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:37.004435 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:37.004688 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:37.005393 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:37.006269 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:37.006494 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:37.006607 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:37.006928 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:37.007354 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:37.007492 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:37.007555 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 04:38:37.007610 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:38:37.014297 16058 raft_consensus.cc:1261] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Refusing update from remote peer 3da1e206bb31410dba2353577e4ed362: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:37.014305 15924 raft_consensus.cc:1261] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Refusing update from remote peer 3da1e206bb31410dba2353577e4ed362: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:37.015484 16262 consensus_queue.cc:1035] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:37.015662 16265 consensus_queue.cc:1035] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:37.024802 16270 mvcc.cc:205] Tried to move back new op lower bound from 7298786578480345088 to 7298786578132180992. Current Snapshot: MvccSnapshot[applied={T|T < 7298786578480345088}]
I20260620 04:38:37.024981 16269 mvcc.cc:205] Tried to move back new op lower bound from 7298786578480345088 to 7298786578132180992. Current Snapshot: MvccSnapshot[applied={T|T < 7298786578480345088}]
I20260620 04:38:37.033107 15779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:37.033291 15779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:37.035562 15779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Sending DeleteTablet for 3 replicas of tablet 7f0c18e99a5a4571b8359186f1f4244d
I20260620 04:38:37.036384 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48954
I20260620 04:38:37.036388 15904 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50652
I20260620 04:38:37.036912 16279 tablet_replica.cc:330] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: stopping tablet replica
I20260620 04:38:37.036892 16038 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42874
I20260620 04:38:37.037222 16279 raft_consensus.cc:2229] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:38:37.037396 16280 tablet_replica.cc:330] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: stopping tablet replica
I20260620 04:38:37.037539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15841
I20260620 04:38:37.037581 16279 raft_consensus.cc:2258] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:37.039690 16279 ts_tablet_manager.cc:1905] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:37.040119 16280 raft_consensus.cc:2229] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:37.040520 16280 raft_consensus.cc:2258] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:37.042091 16280 ts_tablet_manager.cc:1905] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:37.042575 16279 ts_tablet_manager.cc:1918] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:37.042763 16279 log.cc:1196] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/7f0c18e99a5a4571b8359186f1f4244d
I20260620 04:38:37.043143 16279 ts_tablet_manager.cc:1939] T 7f0c18e99a5a4571b8359186f1f4244d P 3da1e206bb31410dba2353577e4ed362: Deleting consensus metadata
I20260620 04:38:37.044337 15764 catalog_manager.cc:4841] TS 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): tablet 7f0c18e99a5a4571b8359186f1f4244d (table test-table [id=e8bc13b68fb14055a47fbf11a442232d]) successfully deleted
W20260620 04:38:37.044914 15763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37433 (error 108)
I20260620 04:38:37.045141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15975
I20260620 04:38:37.046504 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:37.046203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.0.1:50664)
0620 04:38:37.046347 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:37.046417 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 04:38:37.046667 15763 catalog_manager.cc:4584] TS 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f0c18e99a5a4571b8359186f1f4244d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37433: connect: Connection refused (error 111)
I20260620 04:38:37.051684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16109
I20260620 04:38:37.052176 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:37.051891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.0.1:42888)
0620 04:38:37.052050 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:37.052097 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 04:38:37.057557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 15747
I20260620 04:38:57.064090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--webserver_interface=127.0.98.62
--webserver_port=37957
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:38:57.183104 16282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:57.183365 16282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:57.183404 16282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:57.187456 16282 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:38:57.187512 16282 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:38:57.187528 16282 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:38:57.187542 16282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:38:57.187561 16282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:57.187577 16282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:57.187593 16282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:38:57.187650 16282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:38:57.193569 16282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:57.194682 16282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:57.195737 16282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:57.208962 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.208972 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.209005 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:57.209017 16282 server_base.cc:1034] running on GCE node
I20260620 04:38:57.209883 16282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:57.211040 16282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:57.212273 16282 hybrid_clock.cc:648] HybridClock initialized: now 1781930337212195 us; error 90 us; skew 500 ppm
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:57.219909 16282 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:38:57.220968 16282 webserver.cc:460] Webserver started at http://127.0.98.62:37957/ using document root <none> and password file <none>
I20260620 04:38:57.221585 16282 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:57.221688 16282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:57.225628 16282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:38:57.227885 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.229228 16282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:38:57.229393 16282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "b0e08c7c5c89431ca8b9963d75af45c1"
format_stamp: "Formatted at 2026-06-20 04:38:35 on dist-test-slave-zmf4"
server_key: "c78bf204b3520d00a6a164bcfd43d20c"
server_key_iv: "be11caf3f757a32082b1257a6d80bb4f"
server_key_version: "encryptionkey@0"
I20260620 04:38:57.229998 16282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:57.257809 16282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:57.261149 16282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:57.261349 16282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:57.268411 16282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40751
I20260620 04:38:57.268755 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40751 every 8 connection(s)
I20260620 04:38:57.269517 16282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:38:57.271266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16282
I20260620 04:38:57.271736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37433
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=35885
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:38:57.274930 16351 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:38:57.277590 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap starting.
I20260620 04:38:57.300624 16351 log.cc:824] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:57.311125 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:57.311687 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap complete.
I20260620 04:38:57.316421 16351 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:57.316870 16351 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Initialized, Role: FOLLOWER
I20260620 04:38:57.317421 16351 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:57.317570 16351 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:38:57.317668 16351 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:38:57.317790 16351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:38:57.319664 16351 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:57.320019 16351 leader_election.cc:302] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e08c7c5c89431ca8b9963d75af45c1; no voters:
I20260620 04:38:57.320415 16351 leader_election.cc:288] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:38:57.320535 16355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:38:57.320787 16355 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 LEADER]: Becoming Leader. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Running, Role: LEADER
I20260620 04:38:57.321089 16355 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:38:57.321676 16351 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:38:57.322077 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e08c7c5c89431ca8b9963d75af45c1. Latest consensus state: current_term: 2 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:38:57.322187 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:57.322582 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:38:57.322768 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:38:57.322674 16359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:38:57.328276 16359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:38:57.329183 16359 catalog_manager.cc:1259] Loaded cluster ID: ca1fe912743a4e518a3492adb41650c7
I20260620 04:38:57.329255 16359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:38:57.331770 16359 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:38:57.332515 16359 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Loaded TSK: 0
I20260620 04:38:57.355335 16359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Generated new TSK 1
I20260620 04:38:57.355607 16359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:38:57.403745 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:57.404068 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:57.404158 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:57.408984 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:57.409072 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:57.409166 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:38:57.415444 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35885
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:57.416579 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:57.417652 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:57.427421 16378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.427405 16381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.427423 16379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:57.427865 16353 server_base.cc:1034] running on GCE node
I20260620 04:38:57.428313 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:57.429243 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:57.430433 16353 hybrid_clock.cc:648] HybridClock initialized: now 1781930337430421 us; error 33 us; skew 500 ppm
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:57.433307 16353 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:38:57.434374 16353 webserver.cc:460] Webserver started at http://127.0.98.1:35885/ using document root <none> and password file <none>
I20260620 04:38:57.434995 16353 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:57.435071 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:57.439075 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:57.441460 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.442754 16353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:57.442852 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "d05ed3faa13edb5509871afdd9cd4e80"
server_key_iv: "7c27945e5c0ada0b9fb84dc140538469"
server_key_version: "encryptionkey@0"
I20260620 04:38:57.443434 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:57.464843 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:57.468326 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:57.468490 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:57.469323 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:57.470896 16395 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:38:57.476322 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:38:57.476420 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 04:38:57.476519 16353 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:38:57.478200 16353 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:38:57.478271 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:38:57.478554 16395 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap starting.
I20260620 04:38:57.488783 16353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37433
I20260620 04:38:57.488817 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37433 every 8 connection(s)
I20260620 04:38:57.489777 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:38:57.499025 16395 log.cc:824] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:57.499563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16353
I20260620 04:38:57.500034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43215
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=40825
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:57.507055 16395 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:38:57.507498 16395 tablet_bootstrap.cc:492] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap complete.
I20260620 04:38:57.508908 16395 ts_tablet_manager.cc:1397] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20260620 04:38:57.509308 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.492118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:37211)
0620 04:38:57.492823 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.492901 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.493990 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.495211 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.495225 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.495784 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:57.496435 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.496455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.497816 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.497820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.499116 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.499129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.500218 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.500225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.500500 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.501189 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:57.501208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:57.503028 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:57.505667 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.505672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.505683 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.506011 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.506456 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.506459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.506461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.506616 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.507411 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:57.507417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:57.507703 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.507892 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.508270 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":111,"threads_started":1}
I20260620 04:38:57.510020 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.492125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37211 (local address 127.0.98.62:40751)
0620 04:38:57.493029 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.493042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.493078 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:38:57.494467 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.494479 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.494772 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.495055 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.496622 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.497677 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.499319 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.500033 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.500262 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.503236 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.503241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.503254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.503281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.505533 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.506157 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.506166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.506174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.506340 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.506739 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.506745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.506750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.507276 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.507429 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.508082 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.508444 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":177,"threads_started":1}
I20260620 04:38:57.512099 16395 raft_consensus.cc:348] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.512698 16395 raft_consensus.cc:729] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7b4fad258c45f2ba85adfb061f66b5, State: Initialized, Role: FOLLOWER
I20260620 04:38:57.512869 16503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
I20260620 04:38:57.513082 16503 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:57.513248 16395 consensus_queue.cc:260] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.513655 16503 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
I20260620 04:38:57.514638 16395 ts_tablet_manager.cc:1428] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:38:57.516566 16315 ts_manager.cc:194] Registered new tserver with Master: 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:57.519942 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37211
I20260620 04:38:57.522643 16503 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:38:57.530637 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.519282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.0.1:44034)
0620 04:38:57.520042 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.520089 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.520277 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.520781 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.520786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.521215 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:57.521363 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.521371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.522692 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.522696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.523738 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.523752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.524515 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.524532 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.524644 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.525216 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:57.525231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:57.526859 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:57.528881 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.528890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.528895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.529135 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.529442 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.529447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.529451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.529539 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.530003 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:57.530009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:57.530129 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.530265 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.530461 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":117,"threads_started":1}
I20260620 04:38:57.530673 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.519400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.0.98.1:37433)
0620 04:38:57.519796 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.519808 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.520208 (+ 400us) server_negotiation.cc:407] Connection header received
0620 04:38:57.520432 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.520437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.520629 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.520771 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.521489 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.522566 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.523868 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.524401 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.524493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.527023 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.527032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.527045 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.527076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.528747 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.529253 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.529259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.529261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.529346 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.529636 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.529639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.529641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.529875 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.530016 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.530356 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.530525 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":102,"threads_started":1}
I20260620 04:38:57.531608 16437 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0c18e99a5a4571b8359186f1f4244d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44034
I20260620 04:38:57.532631 16512 tablet_replica.cc:330] stopping tablet replica
I20260620 04:38:57.532933 16512 raft_consensus.cc:2229] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:57.533362 16512 raft_consensus.cc:2258] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:57.534483 16512 ts_tablet_manager.cc:1905] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:57.538403 16512 ts_tablet_manager.cc:1918] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:57.538530 16512 log.cc:1196] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal/wals/7f0c18e99a5a4571b8359186f1f4244d
I20260620 04:38:57.538836 16512 ts_tablet_manager.cc:1939] T 7f0c18e99a5a4571b8359186f1f4244d P 3f7b4fad258c45f2ba85adfb061f66b5: Deleting consensus metadata
I20260620 04:38:57.540573 16299 catalog_manager.cc:4841] TS 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433): tablet 7f0c18e99a5a4571b8359186f1f4244d (table test-table [id=e8bc13b68fb14055a47fbf11a442232d]) successfully deleted
W20260620 04:38:57.621587 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:57.621821 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:57.621863 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:57.626133 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:57.626186 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:57.626243 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:38:57.631222 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40825
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:57.632158 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:57.633244 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:57.643163 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.643139 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.643148 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:57.643316 16508 server_base.cc:1034] running on GCE node
I20260620 04:38:57.643774 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:57.644433 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:57.645599 16508 hybrid_clock.cc:648] HybridClock initialized: now 1781930337645567 us; error 41 us; skew 500 ppm
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:57.648597 16508 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:38:57.649503 16508 webserver.cc:460] Webserver started at http://127.0.98.2:40825/ using document root <none> and password file <none>
I20260620 04:38:57.650118 16508 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:57.650228 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:57.654227 16508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:38:57.656615 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.657820 16508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:38:57.657959 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "552b276207fc46d19e2887968912b780"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a06f6836ebe6a373f54cf6dc9b10dc50"
server_key_iv: "a51bd4599d2a77e0f87699dba87b5abe"
server_key_version: "encryptionkey@0"
I20260620 04:38:57.658479 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:57.682479 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:57.685590 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:57.685802 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:57.686393 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:57.687453 16534 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 04:38:57.691537 16534 ts_tablet_manager.cc:1852] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 04:38:57.691947 16534 ts_tablet_manager.cc:1905] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:57.696162 16534 ts_tablet_manager.cc:1918] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:57.696329 16534 log.cc:1196] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/7f0c18e99a5a4571b8359186f1f4244d
I20260620 04:38:57.696830 16534 ts_tablet_manager.cc:1939] T 7f0c18e99a5a4571b8359186f1f4244d P 552b276207fc46d19e2887968912b780: Deleting consensus metadata
I20260620 04:38:57.697366 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 04:38:57.697454 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260620 04:38:57.697552 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:57.697594 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.706887 16508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43215
I20260620 04:38:57.706929 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43215 every 8 connection(s)
I20260620 04:38:57.707703 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:38:57.708429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16508
I20260620 04:38:57.708943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38457
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=39163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:57.722627 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.709617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37637 (local address 127.0.98.62:40751)
0620 04:38:57.709757 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.709761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.710810 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 04:38:57.712085 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.712089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.712149 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.712273 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.713623 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.714238 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.715331 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.715570 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.715840 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.718740 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.718743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.718746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.718772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.720395 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.720939 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.720946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.720948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.721035 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.721407 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.721411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.721413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.721603 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.721717 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.722257 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.722404 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:38:57.723495 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.710048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:37637)
0620 04:38:57.710627 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.710685 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.711888 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.712447 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.712453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.712771 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:57.713463 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.713480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.714360 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.714363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.715204 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.715213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.715869 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.715874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.716069 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.716793 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:57.716811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:57.718551 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:57.720534 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.720541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.720554 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.720824 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.721134 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.721139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.721141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.721297 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.721712 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:57.721719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:57.722004 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.722198 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.722524 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":147,"threads_started":1}
I20260620 04:38:57.724711 16642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
I20260620 04:38:57.724958 16642 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:57.725404 16642 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
I20260620 04:38:57.726440 16315 ts_manager.cc:194] Registered new tserver with Master: 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
I20260620 04:38:57.727170 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37637
W20260620 04:38:57.821441 16644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:38:57.821679 16644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:38:57.821722 16644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:38:57.826243 16644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:38:57.826299 16644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:38:57.826380 16644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:38:57.831578 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39163
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:38:57.832613 16644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:38:57.833598 16644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:38:57.843032 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:57.843091 16644 server_base.cc:1034] running on GCE node
W20260620 04:38:57.843042 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:38:57.843042 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:38:57.843672 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:38:57.844269 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:38:57.845453 16644 hybrid_clock.cc:648] HybridClock initialized: now 1781930337845421 us; error 44 us; skew 500 ppm
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:38:57.848454 16644 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:38:57.849339 16644 webserver.cc:460] Webserver started at http://127.0.98.3:39163/ using document root <none> and password file <none>
I20260620 04:38:57.849993 16644 fs_manager.cc:359] Metadata directory not provided
I20260620 04:38:57.850103 16644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:38:57.854169 16644 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:38:57.856416 16660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.858306 16644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:38:57.858407 16644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "3da1e206bb31410dba2353577e4ed362"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a07276f4910e6535cce60dec27b64e11"
server_key_iv: "e79616b2e5b69ff181f5f7ef0282e073"
server_key_version: "encryptionkey@0"
I20260620 04:38:57.858973 16644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:38:57.871158 16644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:38:57.874480 16644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:38:57.874679 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:38:57.875234 16644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:38:57.876050 16644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:38:57.876097 16644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.876134 16644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:38:57.876220 16644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:38:57.884579 16644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38457
I20260620 04:38:57.884642 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38457 every 8 connection(s)
I20260620 04:38:57.885612 16644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:38:57 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:38:57.895882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16644
I20260620 04:38:57.900779 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.887562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52269 (local address 127.0.98.62:40751)
0620 04:38:57.887734 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.887737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.888935 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 04:38:57.890151 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.890155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.890229 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.890333 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.891638 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.892424 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.893517 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.893736 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.893830 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.896336 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.896342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.896348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.896381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.898443 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.899066 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.899069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.899071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.899150 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.899563 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.899567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.899569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.899814 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.899929 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.900518 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.900654 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 04:38:57.901748 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.887900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:52269)
0620 04:38:57.888776 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.888815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.889930 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.890495 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.890502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.890844 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:38:57.891470 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.891488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.892541 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.892543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.893395 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.893404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.893870 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.893872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.894002 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.894535 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:38:57.894551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:38:57.896118 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:38:57.898583 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.898588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.898603 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.898948 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.899247 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:38:57.899250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:38:57.899252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:38:57.899439 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:38:57.899928 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:38:57.899933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:38:57.900253 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.900427 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.900732 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":556,"threads_started":1}
I20260620 04:38:57.903767 16774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40751
I20260620 04:38:57.904028 16774 heartbeater.cc:464] Registering TS with master...
I20260620 04:38:57.904614 16774 heartbeater.cc:511] Master 127.0.98.62:40751 requested a full tablet report, sending...
I20260620 04:38:57.905748 16315 ts_manager.cc:194] Registered new tserver with Master: 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457)
I20260620 04:38:57.906270 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.896609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.98.62:40751)
0620 04:38:57.896925 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.896930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.897090 (+ 160us) server_negotiation.cc:407] Connection header received
0620 04:38:57.897250 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.897253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.897338 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.897510 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.898583 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.899346 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.900589 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.900912 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.901009 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.902622 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.902628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.902630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.902655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.904450 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.904933 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.904936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.904938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.905023 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.905387 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.905394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.905396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.905620 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.905839 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.905998 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.906115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":88,"threads_started":1}
I20260620 04:38:57.906451 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52269
I20260620 04:38:57.908180 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:38:57.912123 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.908826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.0.98.62:40751)
0620 04:38:57.908963 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.908967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.909047 (+ 80us) server_negotiation.cc:407] Connection header received
0620 04:38:57.909168 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.909170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.909246 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.909325 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:57.909595 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.910099 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.911004 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.911216 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.911309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.911596 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:57.911713 (+ 117us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 04:38:57.911911 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59300: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 04:38:57.912222 16778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59300: token expired
W20260620 04:38:57.912539 16241 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40751)
I20260620 04:38:57.920025 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.912961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.98.62:40751)
0620 04:38:57.913094 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.913100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.913114 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:38:57.913147 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.913149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.913182 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.913385 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:38:57.913938 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.914421 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.915187 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.915412 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.915519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.917049 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:38:57.917052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:38:57.917055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:38:57.917078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:38:57.918463 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.918875 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.918879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.918881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.918973 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:38:57.919233 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:38:57.919237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:38:57.919239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:38:57.919405 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:38:57.919535 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.919761 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.919870 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:38:57.922390 16315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:38:57.925675 16315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:38:57.933810 16437 tablet_service.cc:1467] Processing CreateTablet for tablet 27c78ef72b1148198949d688cb504bd4 (DEFAULT_TABLE table=test-table [id=a5a84f0c02b94117a4ff6edb1601ea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:57.934460 16437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c78ef72b1148198949d688cb504bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:57.937178 16395 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap starting.
I20260620 04:38:57.937945 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.932320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38457 (local address 127.0.0.1:42790)
0620 04:38:57.932435 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.932451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.932537 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.933388 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.933391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.933412 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:57.933536 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.933546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.935194 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.935197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.936458 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.936468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.937507 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.937511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.937608 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.937652 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.937726 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.937806 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":20}
I20260620 04:38:57.938122 16395 tablet_bootstrap.cc:654] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:57.938266 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.932488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.98.3:38457)
0620 04:38:57.932864 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.932875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.932903 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:38:57.932974 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.932978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.933193 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.933399 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:57.933663 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.935059 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.936584 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.937353 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.937508 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.937657 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.937775 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.937866 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":200,"threads_started":1}
I20260620 04:38:57.939669 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.932811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.0.1:48314)
0620 04:38:57.933037 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.933051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.933158 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.934317 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.934320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.934339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:57.934834 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.934840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.936251 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.936255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.937339 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.937348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.939290 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.939293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.939391 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.939412 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.939480 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.939546 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260620 04:38:57.939711 16395 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: No bootstrap required, opened a new log
I20260620 04:38:57.939823 16395 ts_tablet_manager.cc:1397] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:38:57.940078 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.933371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.2:43215)
0620 04:38:57.933810 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.933817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.933834 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:38:57.933891 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.933894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.934067 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.934344 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:57.934952 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.936100 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.937479 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.938327 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.939093 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.939227 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.939824 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.939934 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":153,"threads_started":1}
I20260620 04:38:57.940296 16395 raft_consensus.cc:348] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.940443 16395 raft_consensus.cc:374] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:57.940487 16395 raft_consensus.cc:729] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7b4fad258c45f2ba85adfb061f66b5, State: Initialized, Role: FOLLOWER
I20260620 04:38:57.940711 16395 consensus_queue.cc:260] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.941077 16395 ts_tablet_manager.cc:1428] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:38:57.941334 16708 tablet_service.cc:1467] Processing CreateTablet for tablet 27c78ef72b1148198949d688cb504bd4 (DEFAULT_TABLE table=test-table [id=a5a84f0c02b94117a4ff6edb1601ea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:57.941606 16576 tablet_service.cc:1467] Processing CreateTablet for tablet 27c78ef72b1148198949d688cb504bd4 (DEFAULT_TABLE table=test-table [id=a5a84f0c02b94117a4ff6edb1601ea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:38:57.942437 16576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c78ef72b1148198949d688cb504bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:57.942440 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c78ef72b1148198949d688cb504bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:38:57.947486 16534 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Bootstrap starting.
I20260620 04:38:57.949496 16534 tablet_bootstrap.cc:654] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:57.950196 16534 log.cc:824] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:57.950186 16787 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Bootstrap starting.
I20260620 04:38:57.952315 16787 tablet_bootstrap.cc:654] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Neither blocks nor log segments found. Creating new log.
I20260620 04:38:57.952414 16534 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: No bootstrap required, opened a new log
I20260620 04:38:57.952621 16534 ts_tablet_manager.cc:1397] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:38:57.953639 16787 log.cc:824] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Log is configured to *not* fsync() on all Append() calls
I20260620 04:38:57.955327 16787 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: No bootstrap required, opened a new log
I20260620 04:38:57.955528 16787 ts_tablet_manager.cc:1397] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:38:57.956044 16534 raft_consensus.cc:348] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.956362 16534 raft_consensus.cc:374] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:57.956424 16534 raft_consensus.cc:729] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552b276207fc46d19e2887968912b780, State: Initialized, Role: FOLLOWER
I20260620 04:38:57.956826 16534 consensus_queue.cc:260] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.957485 16642 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
I20260620 04:38:57.957957 16534 ts_tablet_manager.cc:1428] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 04:38:57.958603 16643 tablet.cc:2369] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:57.958985 16787 raft_consensus.cc:348] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.959226 16787 raft_consensus.cc:374] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:38:57.959303 16787 raft_consensus.cc:729] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da1e206bb31410dba2353577e4ed362, State: Initialized, Role: FOLLOWER
I20260620 04:38:57.959707 16787 consensus_queue.cc:260] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.960367 16774 heartbeater.cc:503] Master 127.0.98.62:40751 was elected leader, sending a full tablet report...
I20260620 04:38:57.960644 16787 ts_tablet_manager.cc:1428] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:38:57.977490 16790 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:38:57.977679 16790 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.978861 16790 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:57.983877 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.979161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47421 (local address 127.0.98.3:38457)
0620 04:38:57.979291 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.979298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.979318 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:38:57.979505 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.979511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.979566 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.979667 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:57.980004 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.980807 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.982455 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.983066 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.983191 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.983259 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.983673 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.983746 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:38:57.983877 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.979333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50733 (local address 127.0.98.1:37433)
0620 04:38:57.979490 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:57.979495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:57.979620 (+ 125us) server_negotiation.cc:407] Connection header received
0620 04:38:57.979752 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:57.979755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:57.979827 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:57.979903 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:38:57.980183 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.980962 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.982455 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.983120 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.983209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.983351 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:57.983673 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 04:38:57.983748 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:38:57.983945 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.979067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38457 (local address 127.0.98.2:47421)
0620 04:38:57.979257 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.979273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.979369 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.979711 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.979714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.979734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:57.979850 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.979855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.980945 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.980948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.982319 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.982325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.983261 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.983266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.983510 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.983548 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.983640 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.983769 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":108}
I20260620 04:38:57.984239 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:57.979137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.98.2:50733)
0620 04:38:57.979534 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:57.979546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:38:57.979622 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:38:57.979950 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:38:57.979957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:38:57.979977 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:38:57.980079 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.980083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.981063 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.981066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.982313 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:38:57.982324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:38:57.983261 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:38:57.983266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:38:57.983460 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:57.983546 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:38:57.983640 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:38:57.983772 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":149,"thread_start_us":129,"threads_started":1}
I20260620 04:38:57.984716 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c78ef72b1148198949d688cb504bd4" candidate_uuid: "552b276207fc46d19e2887968912b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da1e206bb31410dba2353577e4ed362" is_pre_election: true
I20260620 04:38:57.984781 16457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c78ef72b1148198949d688cb504bd4" candidate_uuid: "552b276207fc46d19e2887968912b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" is_pre_election: true
I20260620 04:38:57.985055 16728 raft_consensus.cc:2454] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 552b276207fc46d19e2887968912b780 in term 0.
I20260620 04:38:57.985086 16457 raft_consensus.cc:2454] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 552b276207fc46d19e2887968912b780 in term 0.
I20260620 04:38:57.985534 16528 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5, 552b276207fc46d19e2887968912b780; no voters:
I20260620 04:38:57.985814 16790 raft_consensus.cc:2781] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:38:57.985906 16790 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:38:57.985966 16790 raft_consensus.cc:3037] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:57.987041 16790 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:38:57.987463 16790 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [CANDIDATE]: Term 1 election: Requested vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433)
I20260620 04:38:57.987836 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c78ef72b1148198949d688cb504bd4" candidate_uuid: "552b276207fc46d19e2887968912b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da1e206bb31410dba2353577e4ed362"
I20260620 04:38:57.987983 16457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c78ef72b1148198949d688cb504bd4" candidate_uuid: "552b276207fc46d19e2887968912b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
I20260620 04:38:57.987965 16728 raft_consensus.cc:3037] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:57.988128 16457 raft_consensus.cc:3037] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:38:57.989151 16457 raft_consensus.cc:2454] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 552b276207fc46d19e2887968912b780 in term 1.
I20260620 04:38:57.989225 16728 raft_consensus.cc:2454] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 552b276207fc46d19e2887968912b780 in term 1.
I20260620 04:38:57.989480 16528 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5, 552b276207fc46d19e2887968912b780; no voters:
I20260620 04:38:57.989640 16790 raft_consensus.cc:2781] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:38:57.989886 16790 raft_consensus.cc:686] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 1 LEADER]: Becoming Leader. State: Replica: 552b276207fc46d19e2887968912b780, State: Running, Role: LEADER
I20260620 04:38:57.990229 16790 consensus_queue.cc:237] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
W20260620 04:38:57.991053 16504 tablet.cc:2369] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:38:57.993407 16314 catalog_manager.cc:5495] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 552b276207fc46d19e2887968912b780 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "552b276207fc46d19e2887968912b780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } health_report { overall_health: UNKNOWN } } }
I20260620 04:38:58.006042 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:58.002489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.98.2:43215)
0620 04:38:58.002642 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:38:58.002646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:38:58.002660 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:38:58.002723 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:38:58.002725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:38:58.002778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:38:58.002864 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:38:58.003214 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:58.003708 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:58.004587 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:38:58.004800 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:38:58.004893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:38:58.005235 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:38:58.005642 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:38:58.005739 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:38:58.005866 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 04:38:58.005916 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:38:58.011695 16457 raft_consensus.cc:1261] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Refusing update from remote peer 552b276207fc46d19e2887968912b780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:58.011749 16728 raft_consensus.cc:1261] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Refusing update from remote peer 552b276207fc46d19e2887968912b780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:38:58.012401 16790 consensus_queue.cc:1035] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:58.012596 16793 consensus_queue.cc:1035] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:38:58.017758 16797 mvcc.cc:205] Tried to move back new op lower bound from 7298786664488857600 to 7298786664410710016. Current Snapshot: MvccSnapshot[applied={T|T < 7298786664488857600}]
I20260620 04:38:58.018463 16796 mvcc.cc:205] Tried to move back new op lower bound from 7298786664488857600 to 7298786664410710016. Current Snapshot: MvccSnapshot[applied={T|T < 7298786664488857600}]
I20260620 04:38:58.027343 16314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:58.027484 16314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:38:58.029982 16314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Sending DeleteTablet for 3 replicas of tablet 27c78ef72b1148198949d688cb504bd4
I20260620 04:38:58.030647 16708 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c78ef72b1148198949d688cb504bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42790
I20260620 04:38:58.030840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16353
I20260620 04:38:58.031473 16576 tablet_service.cc:1514] Processing DeleteTablet for tablet 27c78ef72b1148198949d688cb504bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:38:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48314
I20260620 04:38:58.031579 16805 tablet_replica.cc:330] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: stopping tablet replica
I20260620 04:38:58.031919 16805 raft_consensus.cc:2229] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:38:58.031952 16806 tablet_replica.cc:330] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: stopping tablet replica
I20260620 04:38:58.032271 16805 raft_consensus.cc:2258] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:58.032338 16806 raft_consensus.cc:2229] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:38:58.032825 16806 raft_consensus.cc:2258] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:38:58.034133 16806 ts_tablet_manager.cc:1905] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:58.034220 16805 ts_tablet_manager.cc:1905] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:38:58.037051 16806 ts_tablet_manager.cc:1918] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:58.037197 16806 log.cc:1196] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/27c78ef72b1148198949d688cb504bd4
I20260620 04:38:58.037540 16806 ts_tablet_manager.cc:1939] T 27c78ef72b1148198949d688cb504bd4 P 552b276207fc46d19e2887968912b780: Deleting consensus metadata
I20260620 04:38:58.037631 16805 ts_tablet_manager.cc:1918] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:38:58.037735 16805 log.cc:1196] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/27c78ef72b1148198949d688cb504bd4
I20260620 04:38:58.038074 16805 ts_tablet_manager.cc:1939] T 27c78ef72b1148198949d688cb504bd4 P 3da1e206bb31410dba2353577e4ed362: Deleting consensus metadata
I20260620 04:38:58.038866 16299 catalog_manager.cc:4841] TS 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): tablet 27c78ef72b1148198949d688cb504bd4 (table test-table [id=a5a84f0c02b94117a4ff6edb1601ea64]) successfully deleted
I20260620 04:38:58.039196 16300 catalog_manager.cc:4841] TS 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): tablet 27c78ef72b1148198949d688cb504bd4 (table test-table [id=a5a84f0c02b94117a4ff6edb1601ea64]) successfully deleted
W20260620 04:38:58.040077 16299 connection.cc:535] client connection to 127.0.98.1:37433 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 04:38:58.040243 16299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:38:58.040750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16508
I20260620 04:38:58.041661 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:38:58.040942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37433 (local address 127.0.0.1:44036)
0620 04:38:58.041489 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 04:38:58.041563 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
I20260620 04:38:58.046576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16644
W20260620 04:38:58.046437 16299 catalog_manager.cc:4584] TS 3f7b4fad258c45f2ba85adfb061f66b5 (127.0.98.1:37433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27c78ef72b1148198949d688cb504bd4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37433: connect: Connection refused (error 111)
I20260620 04:38:58.051913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16282
I20260620 04:39:18.058387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--webserver_interface=127.0.98.62
--webserver_port=37957
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:18.168119 16808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:18.168450 16808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:18.168491 16808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:18.172631 16808 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:39:18.172691 16808 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:39:18.172717 16808 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:39:18.172731 16808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:18.172748 16808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:18.172820 16808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:18.172850 16808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:18.172909 16808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:18.178442 16808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:18.179474 16808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:18.180497 16808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:18.189126 16814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.189277 16808 server_base.cc:1034] running on GCE node
W20260620 04:39:18.189143 16816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.189662 16813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.190440 16808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:18.191592 16808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:18.192813 16808 hybrid_clock.cc:648] HybridClock initialized: now 1781930358192785 us; error 41 us; skew 500 ppm
Jun 20 04:39:18 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:18.208838 16808 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:39:18.210036 16808 webserver.cc:460] Webserver started at http://127.0.98.62:37957/ using document root <none> and password file <none>
I20260620 04:39:18.210584 16808 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:18.210647 16808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:18.215106 16808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:39:18.217782 16823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.219439 16808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:18.219568 16808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "b0e08c7c5c89431ca8b9963d75af45c1"
format_stamp: "Formatted at 2026-06-20 04:38:35 on dist-test-slave-zmf4"
server_key: "c78bf204b3520d00a6a164bcfd43d20c"
server_key_iv: "be11caf3f757a32082b1257a6d80bb4f"
server_key_version: "encryptionkey@0"
I20260620 04:39:18.220131 16808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:18.242009 16808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:18.245330 16808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:18.245486 16808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:18.253036 16808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40751
I20260620 04:39:18.253417 16875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40751 every 8 connection(s)
I20260620 04:39:18.254122 16808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:18.255242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16808
I20260620 04:39:18.256165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37433
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=35885
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:39:18.259069 16876 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:39:18.261663 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap starting.
I20260620 04:39:18.284317 16876 log.cc:824] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:18.299033 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:39:18.299518 16876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Bootstrap complete.
I20260620 04:39:18.305207 16876 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:39:18.305691 16876 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Initialized, Role: FOLLOWER
I20260620 04:39:18.306216 16876 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:39:18.306366 16876 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:18.306425 16876 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:18.306516 16876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:39:18.308911 16876 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:39:18.309372 16876 leader_election.cc:302] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e08c7c5c89431ca8b9963d75af45c1; no voters:
I20260620 04:39:18.309871 16876 leader_election.cc:288] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:39:18.309921 16880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:39:18.310312 16880 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [term 3 LEADER]: Becoming Leader. State: Replica: b0e08c7c5c89431ca8b9963d75af45c1, State: Running, Role: LEADER
I20260620 04:39:18.310640 16880 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } }
I20260620 04:39:18.311261 16876 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:18.311877 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:39:18.311987 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:18.312964 16886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:18.313175 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e08c7c5c89431ca8b9963d75af45c1. Latest consensus state: current_term: 3 leader_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e08c7c5c89431ca8b9963d75af45c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40751 } } }
I20260620 04:39:18.313376 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:18.317943 16886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:18.318702 16886 catalog_manager.cc:1259] Loaded cluster ID: ca1fe912743a4e518a3492adb41650c7
I20260620 04:39:18.318770 16886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:18.320564 16886 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:18.321138 16886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Loaded TSK: 0
I20260620 04:39:18.321224 16886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0e08c7c5c89431ca8b9963d75af45c1: Loaded TSK: 1
F20260620 04:39:18.321871 16886 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7882a5c9ad google::LogMessage::Fail() at ??:0
@ 0x7f7882a5f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7882a5c4de google::LogMessage::Flush() at ??:0
@ 0x7f7882a5da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f788572018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7885718f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f788d3c17bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f788d39b99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f788d39c178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f788d3dddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f788d3fc82b std::function<>::operator()() at ??:0
@ 0x7f788d39bb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f788d39cf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f788d399579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f788d3dd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7884cae3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7884caec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7884cb03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7884c9f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78862e36db start_thread at ??:0
@ 0x7f788155a61f clone at ??:0
W20260620 04:39:18.426927 16878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:18.427174 16878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:18.427245 16878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:18.431367 16878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:18.431420 16878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:18.431507 16878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:18.437023 16878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35885
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:18.438069 16878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:18.439072 16878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:18.449955 16923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.449977 16926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.450093 16878 server_base.cc:1034] running on GCE node
W20260620 04:39:18.450340 16924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.450778 16878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:18.451408 16878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:18.452613 16878 hybrid_clock.cc:648] HybridClock initialized: now 1781930358452586 us; error 48 us; skew 500 ppm
Jun 20 04:39:18 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:18.455583 16878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:39:18.456532 16878 webserver.cc:460] Webserver started at http://127.0.98.1:35885/ using document root <none> and password file <none>
I20260620 04:39:18.457088 16878 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:18.457176 16878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:18.461143 16878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:39:18.463231 16933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.464666 16878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:18.464762 16878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "3f7b4fad258c45f2ba85adfb061f66b5"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "d05ed3faa13edb5509871afdd9cd4e80"
server_key_iv: "7c27945e5c0ada0b9fb84dc140538469"
server_key_version: "encryptionkey@0"
I20260620 04:39:18.465306 16878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:18.482497 16878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:18.485849 16878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:18.486066 16878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:18.486683 16878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:18.487743 16940 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:39:18.491667 16878 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:39:18.491726 16878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:39:18.491773 16878 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:39:18.493544 16878 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:39:18.493611 16878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:18.493850 16940 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap starting.
I20260620 04:39:18.504478 16878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37433
I20260620 04:39:18.504820 17047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37433 every 8 connection(s)
I20260620 04:39:18.505697 16878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:18.513994 16940 log.cc:824] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:18.514935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 16878
I20260620 04:39:18.515427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43215
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=40825
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:39:18.521921 16940 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:39:18.522370 16940 tablet_bootstrap.cc:492] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Bootstrap complete.
I20260620 04:39:18.524024 16940 ts_tablet_manager.cc:1397] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260620 04:39:18.528302 16940 raft_consensus.cc:348] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:18.528990 16940 raft_consensus.cc:729] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7b4fad258c45f2ba85adfb061f66b5, State: Initialized, Role: FOLLOWER
I20260620 04:39:18.529559 16940 consensus_queue.cc:260] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:18.530802 16940 ts_tablet_manager.cc:1428] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
W20260620 04:39:18.543497 17050 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:39:18.507862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:56501)
0620 04:39:18.508448 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.508514 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:18.509637 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:18.542609 (+ 32972us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":169,"threads_started":1}
W20260620 04:39:18.544094 17048 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40751 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40751: Client connection negotiation failed: client connection to 127.0.98.62:40751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:39:18.544849 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.544585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:43099)
0620 04:39:18.544718 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.544787 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 04:39:18.545557 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.545341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:55733)
0620 04:39:18.545451 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.545500 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 04:39:18.545787 17048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:39:18.632937 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:18.633213 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:18.633260 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:18.637612 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:18.637682 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:18.637741 17052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:18.643009 17052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40825
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:18.644088 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:18.645256 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:18.654960 17058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.654997 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.654968 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.655189 17052 server_base.cc:1034] running on GCE node
I20260620 04:39:18.655663 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:18.656327 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:18.657500 17052 hybrid_clock.cc:648] HybridClock initialized: now 1781930358657477 us; error 36 us; skew 500 ppm
Jun 20 04:39:18 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:18.660423 17052 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:39:18.661294 17052 webserver.cc:460] Webserver started at http://127.0.98.2:40825/ using document root <none> and password file <none>
I20260620 04:39:18.661868 17052 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:18.661962 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:18.665879 17052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:39:18.667977 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.669286 17052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:18.669411 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "552b276207fc46d19e2887968912b780"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a06f6836ebe6a373f54cf6dc9b10dc50"
server_key_iv: "a51bd4599d2a77e0f87699dba87b5abe"
server_key_version: "encryptionkey@0"
I20260620 04:39:18.670002 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:18.680595 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:18.683353 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:18.683558 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:18.684175 17052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:18.685014 17052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:18.685083 17052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.685151 17052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:18.685204 17052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.694917 17052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43215
I20260620 04:39:18.694939 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43215 every 8 connection(s)
I20260620 04:39:18.695892 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:18.699985 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.698302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:50599)
0620 04:39:18.698770 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.698881 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
W20260620 04:39:18.700484 17182 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40751 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40751: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
I20260620 04:39:18.701066 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.700863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:41787)
0620 04:39:18.700970 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.701014 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:39:18.701711 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.701511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:53263)
0620 04:39:18.701623 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.701666 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 04:39:18.701964 17182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:39:18.704401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17052
I20260620 04:39:18.704802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38457
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40751
--webserver_port=39163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:18.810883 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:18.811156 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:18.811233 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:18.815313 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:18.815389 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:18.815476 17185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:18.820501 17185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39163
--tserver_master_addrs=127.0.98.62:40751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:18.821465 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:18.822420 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:39:18.831673 17185 server_base.cc:1034] running on GCE node
W20260620 04:39:18.831599 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.831599 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:18.831619 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:18.832235 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:18.832796 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:18.833976 17185 hybrid_clock.cc:648] HybridClock initialized: now 1781930358833955 us; error 33 us; skew 500 ppm
Jun 20 04:39:18 dist-test-slave-zmf4 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:18.836726 17185 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:39:18.837594 17185 webserver.cc:460] Webserver started at http://127.0.98.3:39163/ using document root <none> and password file <none>
I20260620 04:39:18.838085 17185 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:18.838141 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:18.841693 17185 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:39:18.843578 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.844750 17185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 04:39:18.844880 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "3da1e206bb31410dba2353577e4ed362"
format_stamp: "Formatted at 2026-06-20 04:38:36 on dist-test-slave-zmf4"
server_key: "a07276f4910e6535cce60dec27b64e11"
server_key_iv: "e79616b2e5b69ff181f5f7ef0282e073"
server_key_version: "encryptionkey@0"
I20260620 04:39:18.845459 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:18.859856 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:18.862900 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:18.863039 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:18.863511 17185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:18.864296 17185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:18.864341 17185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.864379 17185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:18.864394 17185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:18.873814 17185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38457
I20260620 04:39:18.873850 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38457 every 8 connection(s)
I20260620 04:39:18.874760 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:18.878456 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.876777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:37485)
0620 04:39:18.877264 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.877374 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
W20260620 04:39:18.878922 17314 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40751 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40751: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
I20260620 04:39:18.879485 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.879256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:36121)
0620 04:39:18.879376 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.879434 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 04:39:18.880237 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:18.879993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:46243)
0620 04:39:18.880118 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:18.880165 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 04:39:18.880453 17314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:39:18.880730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17185
I20260620 04:39:19.546892 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:19.546416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:47867)
0620 04:39:19.546727 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:19.546779 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260620 04:39:19.702915 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:19.702494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:43927)
0620 04:39:19.702753 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:19.702804 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":66,"threads_started":1}
I20260620 04:39:19.881338 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:19.880935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:45557)
0620 04:39:19.881168 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:19.881238 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260620 04:39:20.007222 17321 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:20.007472 17321 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:20.008675 17321 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
Jun 20 04:39:20 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 04:39:20 dist-test-slave-zmf4 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 04:39:20.023317 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.008887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.98.1:52555)
0620 04:39:20.009010 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:20.009029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:20.009144 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:20.010883 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:20.010889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:20.011449 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:20.012046 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:20.012070 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.013673 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.013680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:20.014749 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:20.014759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.015535 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.015538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:20.015686 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:20.016425 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:20.016449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:20.018077 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:20.020541 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:20.020546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:20.020560 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:20.021048 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:20.021453 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:20.021459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:20.021689 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:20.022435 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:20.022442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:20.022745 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 04:39:20.022948 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 04:39:20.023175 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":48,"spinlock_wait_cycles":59520}
I20260620 04:39:20.023408 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.009072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52555 (local address 127.0.98.2:43215)
0620 04:39:20.009421 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:20.009450 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:20.009498 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:39:20.010223 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:20.010229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:20.010529 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:20.010837 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:20.012251 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.013460 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.014881 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.015383 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.015575 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:20.018289 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:20.018295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:20.018324 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:20.018355 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:20.020425 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021168 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:20.021175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:20.021180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:20.021334 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021800 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:20.021804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:20.021806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:20.022294 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:20.022430 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:20.023054 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 04:39:20.023264 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
I20260620 04:39:20.024109 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.008887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38457 (local address 127.0.98.1:39265)
0620 04:39:20.009375 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:20.009389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:20.009502 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:20.010885 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:20.010891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:20.011451 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:20.012046 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:20.012070 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.013674 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.013677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:20.014747 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:20.014758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.015494 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.015498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:20.015689 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:20.016465 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:20.016478 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:20.018475 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:20.020692 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:20.020695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:20.020697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:20.021048 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:20.021453 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:20.021459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:20.021682 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:20.023522 (+ 1840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:20.023525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:20.023655 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 04:39:20.023794 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 04:39:20.023933 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":373,"spinlock_wait_cycles":9984,"thread_start_us":73,"threads_started":1}
I20260620 04:39:20.024510 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.009049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39265 (local address 127.0.98.3:38457)
0620 04:39:20.009471 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:20.009493 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:20.009534 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:39:20.010193 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:20.010198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:20.010529 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:20.010837 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:20.012244 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.013448 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.014912 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:20.015365 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:20.015516 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:20.018661 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:20.018669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:20.018689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:20.018715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:20.020588 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021168 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:20.021175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:20.021180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:20.021334 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:20.021800 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:20.021804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:20.021806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:20.022294 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:20.022430 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:20.023945 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0620 04:39:20.024372 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":114,"threads_started":1}
W20260620 04:39:20.026353 16934 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
W20260620 04:39:20.026520 16935 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
I20260620 04:39:20.026630 16935 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5; no voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780
I20260620 04:39:20.026925 17321 raft_consensus.cc:2726] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:39:20.548254 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.547785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:54467)
0620 04:39:20.548047 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:20.548100 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260620 04:39:20.704183 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.703773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:49455)
0620 04:39:20.704041 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:20.704097 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260620 04:39:20.882486 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:20.882085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:59769)
0620 04:39:20.882341 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:20.882395 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
I20260620 04:39:21.549541 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:21.549100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:55957)
0620 04:39:21.549392 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:21.549447 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
I20260620 04:39:21.705418 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:21.704953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:43791)
0620 04:39:21.705250 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:21.705309 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260620 04:39:21.883754 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:21.883331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:47187)
0620 04:39:21.883579 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:21.883630 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260620 04:39:22.080721 17331 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:22.080849 17331 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:22.081229 17331 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
W20260620 04:39:22.081992 16934 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
W20260620 04:39:22.082104 16935 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
I20260620 04:39:22.082219 16935 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5; no voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780
I20260620 04:39:22.082402 17331 raft_consensus.cc:2726] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:39:22.551013 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:22.550383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:59641)
0620 04:39:22.550732 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:22.550820 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260620 04:39:22.706674 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:22.706242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:53169)
0620 04:39:22.706516 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:22.706578 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260620 04:39:22.884946 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:22.884536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:54411)
0620 04:39:22.884803 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:22.884862 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260620 04:39:23.552392 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:23.551908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:42057)
0620 04:39:23.552234 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:23.552297 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260620 04:39:23.708045 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:23.707562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:39053)
0620 04:39:23.707855 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:23.707921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260620 04:39:23.886102 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:23.885679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:59841)
0620 04:39:23.885958 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:23.886012 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260620 04:39:24.290911 17338 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:24.291064 17338 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:24.291527 17338 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
W20260620 04:39:24.292320 16935 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
W20260620 04:39:24.292419 16934 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
I20260620 04:39:24.292541 16934 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5; no voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780
I20260620 04:39:24.292677 17338 raft_consensus.cc:2726] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:39:24.553633 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:24.553175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:48907)
0620 04:39:24.553460 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:24.553532 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
I20260620 04:39:24.709316 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:24.708846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:37721)
0620 04:39:24.709138 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:24.709205 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260620 04:39:24.887405 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:24.886984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:48225)
0620 04:39:24.887227 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:24.887276 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260620 04:39:25.554879 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:25.554450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:34157)
0620 04:39:25.554717 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:25.554782 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260620 04:39:25.710563 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:25.710102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:48275)
0620 04:39:25.710413 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:25.710471 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":87,"threads_started":1}
I20260620 04:39:25.888722 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:25.888245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:33617)
0620 04:39:25.888571 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:25.888626 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":68,"threads_started":1}
I20260620 04:39:26.188534 17345 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:26.188688 17345 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:26.189146 17345 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
W20260620 04:39:26.189813 16935 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
W20260620 04:39:26.189921 16934 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
I20260620 04:39:26.190008 16934 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5; no voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780
I20260620 04:39:26.190148 17345 raft_consensus.cc:2726] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:39:26.556309 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:26.555782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:47321)
0620 04:39:26.556065 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:26.556127 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260620 04:39:26.711897 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:26.711440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:43875)
0620 04:39:26.711724 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:26.711802 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260620 04:39:26.889971 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:26.889520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:54677)
0620 04:39:26.889811 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:26.889873 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260620 04:39:27.557646 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:27.557102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:49971)
0620 04:39:27.557482 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:27.557541 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":196,"threads_started":1}
I20260620 04:39:27.713187 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:27.712754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:50365)
0620 04:39:27.713012 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:27.713067 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260620 04:39:27.891528 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:27.890859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:60669)
0620 04:39:27.891364 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:27.891430 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":83,"threads_started":1}
I20260620 04:39:28.559329 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:28.558922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:56779)
0620 04:39:28.559184 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:28.559232 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260620 04:39:28.714443 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:28.713962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:59027)
0620 04:39:28.714231 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:28.714286 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
I20260620 04:39:28.892856 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:28.892431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:35635)
0620 04:39:28.892674 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:28.892731 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260620 04:39:29.560814 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:29.560302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:52957)
0620 04:39:29.560599 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:29.560654 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260620 04:39:29.715615 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:29.715195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:41997)
0620 04:39:29.715454 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:29.715512 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260620 04:39:29.894140 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:29.893687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:59641)
0620 04:39:29.893981 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:29.894044 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260620 04:39:30.562103 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:30.561674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:33309)
0620 04:39:30.561945 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:30.561998 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260620 04:39:30.716818 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:30.716363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:41575)
0620 04:39:30.716652 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:30.716708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260620 04:39:30.895324 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:30.894891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:51555)
0620 04:39:30.895164 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:30.895223 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260620 04:39:31.563387 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:31.562904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:48867)
0620 04:39:31.563190 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:31.563271 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260620 04:39:31.718248 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:31.717848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:33795)
0620 04:39:31.718097 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:31.718154 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260620 04:39:31.896914 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:31.896467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:44899)
0620 04:39:31.896735 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:31.896805 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260620 04:39:32.564807 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:32.564355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:49037)
0620 04:39:32.564636 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:32.564696 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260620 04:39:32.719815 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:32.719373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:59129)
0620 04:39:32.719658 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:32.719717 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260620 04:39:32.898219 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:32.897775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.3:55897)
0620 04:39:32.898065 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:32.898123 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260620 04:39:33.278249 17367 raft_consensus.cc:482] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:33.278389 17367 raft_consensus.cc:504] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da1e206bb31410dba2353577e4ed362" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38457 } } peers { permanent_uuid: "552b276207fc46d19e2887968912b780" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } } peers { permanent_uuid: "3f7b4fad258c45f2ba85adfb061f66b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37433 } }
I20260620 04:39:33.278759 17367 leader_election.cc:288] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457), 552b276207fc46d19e2887968912b780 (127.0.98.2:43215)
W20260620 04:39:33.279531 16934 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 552b276207fc46d19e2887968912b780 (127.0.98.2:43215): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
W20260620 04:39:33.279659 16935 leader_election.cc:341] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3da1e206bb31410dba2353577e4ed362 (127.0.98.3:38457): Not found: Tablet not found: 27c78ef72b1148198949d688cb504bd4
I20260620 04:39:33.279742 16935 leader_election.cc:302] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f7b4fad258c45f2ba85adfb061f66b5; no voters: 3da1e206bb31410dba2353577e4ed362, 552b276207fc46d19e2887968912b780
I20260620 04:39:33.279906 17367 raft_consensus.cc:2726] T 27c78ef72b1148198949d688cb504bd4 P 3f7b4fad258c45f2ba85adfb061f66b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:39:33.566391 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:33.565943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.1:60771)
0620 04:39:33.566238 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:33.566295 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260620 04:39:33.721208 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:33.720750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40751 (local address 127.0.98.2:60801)
0620 04:39:33.721042 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:33.721101 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260620 04:39:33.885649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 16878
I20260620 04:39:33.891523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17052
I20260620 04:39:33.897047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17185
2026-06-20T04:39:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59471 ms)
[----------] 30 tests from SecurityITest (245776 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:33 dist-test-slave-zmf4 krb5kdc[17372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:33 dist-test-slave-zmf4 krb5kdc[17372](info): set up 2 sockets
Jun 20 04:39:33 dist-test-slave-zmf4 krb5kdc[17372](info): commencing operation
krb5kdc: starting...
W20260620 04:39:34.955983 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:39:35.075433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:35.182874 17385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:35.183185 17385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:35.183255 17385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:35.187373 17385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:35.187453 17385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:35.187489 17385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:35.187527 17385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:35.187562 17385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:35.192785 17385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:35.193848 17385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:35.194833 17385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:35.203488 17393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.203488 17391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.203649 17390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:35.203666 17385 server_base.cc:1034] running on GCE node
I20260620 04:39:35.204411 17385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:35.205452 17385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:35.206681 17385 hybrid_clock.cc:648] HybridClock initialized: now 1781930375206639 us; error 62 us; skew 500 ppm
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:35.212921 17385 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:39:35.213832 17385 webserver.cc:460] Webserver started at http://127.0.98.62:37599/ using document root <none> and password file <none>
I20260620 04:39:35.214422 17385 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:35.214526 17385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:35.214776 17385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:35.217121 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "eb5a110a545247c8b081501aa2bd8cc9"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "5d877dbcd518ff557795b1cc66f8c756"
server_key_iv: "b1e8bb76d5c3c8daef126c8855bebe3c"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.217700 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb5a110a545247c8b081501aa2bd8cc9"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "5d877dbcd518ff557795b1cc66f8c756"
server_key_iv: "b1e8bb76d5c3c8daef126c8855bebe3c"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.221305 17385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:39:35.223776 17400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.225106 17385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:35.225265 17385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "eb5a110a545247c8b081501aa2bd8cc9"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "5d877dbcd518ff557795b1cc66f8c756"
server_key_iv: "b1e8bb76d5c3c8daef126c8855bebe3c"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.225378 17385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:35.248575 17385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:35.251668 17385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:35.251868 17385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:35.258783 17385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42923
I20260620 04:39:35.258809 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42923 every 8 connection(s)
I20260620 04:39:35.259665 17385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:35.261149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17385
I20260620 04:39:35.261276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:35.261579 392 external_mini_cluster.cc:1428] Setting key 77ad5796ff32d57f5dbf9be64cd2ed7c
I20260620 04:39:35.262641 17453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:35.269150 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: Bootstrap starting.
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:35.271552 17453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:35.272480 17453 log.cc:824] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:35.274525 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: No bootstrap required, opened a new log
I20260620 04:39:35.276549 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:35.262908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.0.98.62:42923)
0620 04:39:35.263304 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:35.263322 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:35.263361 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:39:35.264167 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:35.264209 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:35.264594 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:35.264932 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:35.265618 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.267081 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.268020 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.268696 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.268853 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:35.271447 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:35.271456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:35.271473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:35.271509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:35.273343 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.273751 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.273756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.273762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.273891 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.274220 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.274223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.274228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.274690 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:35.274870 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:35.275142 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 04:39:35.275421 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260620 04:39:35.278198 17453 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 04:39:35.278426 17453 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:35.278545 17453 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5a110a545247c8b081501aa2bd8cc9, State: Initialized, Role: FOLLOWER
I20260620 04:39:35.278967 17453 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 04:39:35.279083 17453 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:35.279121 17453 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:35.279188 17453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:35.279942 17453 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 04:39:35.280257 17453 leader_election.cc:302] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5a110a545247c8b081501aa2bd8cc9; no voters:
I20260620 04:39:35.280517 17453 leader_election.cc:288] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:35.280618 17458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:35.280867 17458 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [term 1 LEADER]: Becoming Leader. State: Replica: eb5a110a545247c8b081501aa2bd8cc9, State: Running, Role: LEADER
I20260620 04:39:35.281167 17458 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } }
I20260620 04:39:35.281579 17453 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:35.282758 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5a110a545247c8b081501aa2bd8cc9. Latest consensus state: current_term: 1 leader_uuid: "eb5a110a545247c8b081501aa2bd8cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 04:39:35.282927 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:35.283238 17459 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5a110a545247c8b081501aa2bd8cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5a110a545247c8b081501aa2bd8cc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42923 } } }
I20260620 04:39:35.283316 17459 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9 [sys.catalog]: This master's current role is: LEADER
W20260620 04:39:35.286463 17473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:39:35.286526 17473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:39:35.286608 17468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:35.287357 17468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:35.292707 17468 catalog_manager.cc:1347] Generated new cluster ID: 0f03e305ac3e4e0393c16349b0a2f3c7
I20260620 04:39:35.292779 17468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:35.334730 17468 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:35.336112 17468 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:35.344573 17468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: Generated new TSK 0
I20260620 04:39:35.345225 17468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:39:35.375303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:35.495507 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:35.495738 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:35.495780 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:35.499652 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:35.499704 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:35.499759 17479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:35.504926 17479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:35.505818 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:35.506733 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:35.516106 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:35.516202 17479 server_base.cc:1034] running on GCE node
W20260620 04:39:35.516307 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.516307 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:35.516871 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:35.517475 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:35.518684 17479 hybrid_clock.cc:648] HybridClock initialized: now 1781930375518658 us; error 36 us; skew 500 ppm
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:35.521623 17479 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:39:35.522509 17479 webserver.cc:460] Webserver started at http://127.0.98.1:38661/ using document root <none> and password file <none>
I20260620 04:39:35.523052 17479 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:35.523144 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:35.523370 17479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:35.525375 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1e15eb4834f4a5288c8ede0c2c3de20"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8d3b0f0966f7326f2598b44fb7bf9b57"
server_key_iv: "fc430be8c61b0263e71f43649fffbf84"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.525939 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1e15eb4834f4a5288c8ede0c2c3de20"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8d3b0f0966f7326f2598b44fb7bf9b57"
server_key_iv: "fc430be8c61b0263e71f43649fffbf84"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.529695 17479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:39:35.532119 17494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.533356 17479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:35.533499 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "e1e15eb4834f4a5288c8ede0c2c3de20"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8d3b0f0966f7326f2598b44fb7bf9b57"
server_key_iv: "fc430be8c61b0263e71f43649fffbf84"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.533613 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:35.553323 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:35.556424 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:35.556620 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:35.557185 17479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:35.557997 17479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:35.558076 17479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.558147 17479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:35.558202 17479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.567399 17479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45025
I20260620 04:39:35.567421 17607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45025 every 8 connection(s)
I20260620 04:39:35.568302 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:35.571568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17479
I20260620 04:39:35.571699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:35.572010 392 external_mini_cluster.cc:1428] Setting key a71125234cdd18450fb29e659d95b17d
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:39:35.582507 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:35.570170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52305 (local address 127.0.98.62:42923)
0620 04:39:35.570305 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:35.570310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:35.571041 (+ 731us) server_negotiation.cc:407] Connection header received
0620 04:39:35.572293 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:35.572296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:35.572348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:35.572426 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:35.573749 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.574331 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.575794 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.576031 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.576142 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:35.578704 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:35.578708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:35.578710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:35.578741 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:35.580288 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.580792 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.580797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.580799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.580884 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.581269 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.581274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.581275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.581457 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:35.581567 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:35.582199 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 04:39:35.582336 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:39:35.583516 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:35.570446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.1:52305)
0620 04:39:35.570900 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:35.570940 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:35.571990 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:35.572608 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:35.572615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:35.573041 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:35.573598 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:35.573618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.574469 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.574476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:35.575665 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:35.575675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.576212 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.576217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:35.576375 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:35.577034 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:35.577050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:35.578537 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:35.580410 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:35.580416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:35.580430 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:35.580685 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:35.580978 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:35.581021 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:35.581023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:35.581172 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:35.581563 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:35.581569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:35.581924 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 04:39:35.582146 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 04:39:35.582454 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260620 04:39:35.584870 17608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 04:39:35.585136 17608 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:35.585721 17608 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
I20260620 04:39:35.587507 17417 ts_manager.cc:194] Registered new tserver with Master: e1e15eb4834f4a5288c8ede0c2c3de20 (127.0.98.1:45025)
I20260620 04:39:35.589278 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:39:35.599819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:35.707163 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:35.707408 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:35.707450 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:35.711309 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:35.711364 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:35.711419 17613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:35.716341 17613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:35.717232 17613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:35.718109 17613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:35.727752 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.727761 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.727806 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:35.728245 17613 server_base.cc:1034] running on GCE node
I20260620 04:39:35.728627 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:35.729206 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:35.730376 17613 hybrid_clock.cc:648] HybridClock initialized: now 1781930375730343 us; error 43 us; skew 500 ppm
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:35.733163 17613 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:39:35.734004 17613 webserver.cc:460] Webserver started at http://127.0.98.2:44339/ using document root <none> and password file <none>
I20260620 04:39:35.734541 17613 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:35.734629 17613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:35.734850 17613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:35.736927 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "094ee67b1d2746079a5ca23facbd7559"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "68955e61fd940993aa635daa11f27b74"
server_key_iv: "6695de08756b7d45515dfd1efbcf1714"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.737474 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "094ee67b1d2746079a5ca23facbd7559"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "68955e61fd940993aa635daa11f27b74"
server_key_iv: "6695de08756b7d45515dfd1efbcf1714"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.741151 17613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:39:35.743505 17628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.744854 17613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:39:35.744993 17613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "094ee67b1d2746079a5ca23facbd7559"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "68955e61fd940993aa635daa11f27b74"
server_key_iv: "6695de08756b7d45515dfd1efbcf1714"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.745100 17613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:35.772331 17613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:35.775910 17613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:35.776108 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:35.776711 17613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:35.777540 17613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:35.777611 17613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.777679 17613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:35.777729 17613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.787094 17613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34105
I20260620 04:39:35.787113 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34105 every 8 connection(s)
I20260620 04:39:35.787992 17613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:35.796321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17613
I20260620 04:39:35.796437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:35.796820 392 external_mini_cluster.cc:1428] Setting key 42bf744bd7be23b9804977803bd8515e
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:35.802351 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:35.789957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57473 (local address 127.0.98.62:42923)
0620 04:39:35.790125 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:35.790129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:35.791119 (+ 990us) server_negotiation.cc:407] Connection header received
0620 04:39:35.791992 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:35.791995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:35.792053 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:35.792134 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:35.793253 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.793777 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.794881 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.795113 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.795225 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:35.798109 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:35.798114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:35.798120 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:35.798153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:35.799968 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.800684 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.800687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.800688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.800766 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:35.801165 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:35.801167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:35.801168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:35.801335 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:35.801424 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:35.801972 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 04:39:35.802110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:39:35.802910 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:35.790303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.2:57473)
0620 04:39:35.790962 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:35.791000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:35.791790 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:35.792330 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:35.792336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:35.792671 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:35.793110 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:35.793123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.793910 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.793921 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:35.794758 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:35.794767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:35.795259 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:35.795263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:35.795434 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:35.795971 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:35.795987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:35.797898 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:35.800112 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:35.800121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:35.800143 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:35.800565 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:35.800872 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:35.800876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:35.800880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:35.801064 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:35.801440 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:35.801448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:35.801715 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 04:39:35.801865 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 04:39:35.802096 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":109,"threads_started":1}
I20260620 04:39:35.804044 17742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 04:39:35.804315 17742 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:35.804764 17742 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
I20260620 04:39:35.805792 17417 ts_manager.cc:194] Registered new tserver with Master: 094ee67b1d2746079a5ca23facbd7559 (127.0.98.2:34105)
I20260620 04:39:35.806371 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:39:35.825100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:35.934978 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:35.935204 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:35.935245 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:35.939435 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:35.939491 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:35.939550 17747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:35.944947 17747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:35.945955 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:35.946909 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:35.956143 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.956151 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:35.956249 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:35.956387 17747 server_base.cc:1034] running on GCE node
I20260620 04:39:35.956894 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:35.957515 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:35.958724 17747 hybrid_clock.cc:648] HybridClock initialized: now 1781930375958691 us; error 44 us; skew 500 ppm
Jun 20 04:39:35 dist-test-slave-zmf4 krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:35.961939 17747 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:39:35.962883 17747 webserver.cc:460] Webserver started at http://127.0.98.3:33735/ using document root <none> and password file <none>
I20260620 04:39:35.963474 17747 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:35.963574 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:35.963819 17747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:35.966001 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "56dfe04b97324dde8910c0519edc2b11"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8171a985662ef760206612f0e9aa32c7"
server_key_iv: "a039c767b5e8ee213bb7b87f20994702"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.966595 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56dfe04b97324dde8910c0519edc2b11"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8171a985662ef760206612f0e9aa32c7"
server_key_iv: "a039c767b5e8ee213bb7b87f20994702"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.970518 17747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:39:35.972929 17762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:35.974076 17747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:35.974212 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "56dfe04b97324dde8910c0519edc2b11"
format_stamp: "Formatted at 2026-06-20 04:39:35 on dist-test-slave-zmf4"
server_key: "8171a985662ef760206612f0e9aa32c7"
server_key_iv: "a039c767b5e8ee213bb7b87f20994702"
server_key_version: "encryptionkey@0"
I20260620 04:39:35.974321 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:36.010921 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:36.014191 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:36.014413 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:36.015012 17747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:36.015827 17747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:36.015902 17747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.015977 17747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:36.016031 17747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.025930 17747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45425
I20260620 04:39:36.025964 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45425 every 8 connection(s)
I20260620 04:39:36.026775 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:36.031411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17747
I20260620 04:39:36.031581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:36.031997 392 external_mini_cluster.cc:1428] Setting key ab5b83af4c04dd4a0a4c38dac38018ed
Jun 20 04:39:36 dist-test-slave-zmf4 krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:36.040903 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.028549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58139 (local address 127.0.98.62:42923)
0620 04:39:36.028697 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.028702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.029737 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 04:39:36.030626 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.030630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.030687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.030777 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:36.031979 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.032776 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.034109 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.034345 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.034457 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.037033 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:36.037037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:36.037040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:36.037067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:36.038607 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:36.039130 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:36.039133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:36.039134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:36.039225 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:36.039607 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:36.039611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:36.039612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:36.039792 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:36.039877 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.040665 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.040776 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:39:36.041924 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.028990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42923 (local address 127.0.98.3:58139)
0620 04:39:36.029535 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.029615 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.030430 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.030937 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.030943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.031289 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:36.031841 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.031855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.032883 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.032891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.033991 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.034000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.034492 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.034495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.034656 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.035243 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:36.035266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:36.036835 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:36.038738 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:36.038743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:36.038755 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:36.039021 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:36.039342 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:36.039346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:36.039348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:36.039489 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:36.039914 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:36.039919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:36.040159 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.040507 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.041070 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":116,"threads_started":1}
I20260620 04:39:36.043087 17876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42923
I20260620 04:39:36.043335 17876 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:36.043804 17876 heartbeater.cc:511] Master 127.0.98.62:42923 requested a full tablet report, sending...
I20260620 04:39:36.044937 17417 ts_manager.cc:194] Registered new tserver with Master: 56dfe04b97324dde8910c0519edc2b11 (127.0.98.3:45425)
I20260620 04:39:36.045660 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58139
I20260620 04:39:36.045957 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:36.055959 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.047934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.0.98.62:42923)
0620 04:39:36.048081 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.048087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.048272 (+ 185us) server_negotiation.cc:407] Connection header received
0620 04:39:36.048422 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.048425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.048498 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.048585 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:36.049296 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.049904 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.050789 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.051131 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.051243 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.052880 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:36.052883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:36.052885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:36.052909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:36.054332 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:36.054745 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:36.054748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:36.054749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:36.054816 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:36.055096 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:36.055099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:36.055100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:36.055345 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:36.055460 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.055746 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.055847 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:39:36.059019 17417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:36.062032 17417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:36.075876 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45025 (local address 127.0.0.1:38602)
0620 04:39:36.070434 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.070478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.070602 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.071380 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.071386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.071410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:36.071547 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.071555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.073020 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.073026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.074250 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.074260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.075327 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.075331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.075434 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.075472 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.075592 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.075704 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":76,"thread_start_us":88,"threads_started":1}
I20260620 04:39:36.075911 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.0.98.1:45025)
0620 04:39:36.070894 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.070909 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.070931 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:39:36.071017 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.071022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.071174 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.071704 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:36.071883 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.072896 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.074399 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.075195 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.075301 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.075450 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.075635 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.075730 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":84,"threads_started":1}
I20260620 04:39:36.076458 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45425 (local address 127.0.0.1:52416)
0620 04:39:36.071413 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.071428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.071531 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.072249 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.072252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.072269 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:36.072366 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.072371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.073827 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.073833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.074877 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.074886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.075993 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.076000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.076125 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.076152 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.076263 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.076339 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":26,"thread_start_us":133,"threads_started":1}
I20260620 04:39:36.076741 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.0.98.2:34105)
0620 04:39:36.071492 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.071504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.071524 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:39:36.071600 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.071606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.071770 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.071914 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:36.072225 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.073484 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.075015 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.075789 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.075899 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.076040 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.076519 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.076606 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":93,"threads_started":1}
I20260620 04:39:36.076987 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34105 (local address 127.0.0.1:51624)
0620 04:39:36.070828 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.070841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.070924 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.071968 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.071972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.071995 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:36.072086 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.072092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.073635 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.073639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.074875 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.074885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.076103 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.076107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.076184 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.076255 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.076329 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.076404 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":32,"thread_start_us":105,"threads_started":1}
I20260620 04:39:36.077304 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.070795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.0.98.3:45425)
0620 04:39:36.071194 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.071207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.071629 (+ 422us) server_negotiation.cc:407] Connection header received
0620 04:39:36.071701 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.071705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.071869 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.072099 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:36.072488 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.073685 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.074998 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.075706 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.075812 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.075935 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.076983 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.077107 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":112,"threads_started":1}
I20260620 04:39:36.078943 17542 tablet_service.cc:1467] Processing CreateTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 (DEFAULT_TABLE table=test-table [id=f5724862ea3148cb904962f73d559e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:36.079546 17810 tablet_service.cc:1467] Processing CreateTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 (DEFAULT_TABLE table=test-table [id=f5724862ea3148cb904962f73d559e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:36.079965 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:36.080433 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:36.082455 17676 tablet_service.cc:1467] Processing CreateTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 (DEFAULT_TABLE table=test-table [id=f5724862ea3148cb904962f73d559e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:36.083384 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:36.086877 17893 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Bootstrap starting.
I20260620 04:39:36.087507 17894 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Bootstrap starting.
I20260620 04:39:36.089099 17893 tablet_bootstrap.cc:654] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:36.089519 17894 tablet_bootstrap.cc:654] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:36.090055 17895 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Bootstrap starting.
I20260620 04:39:36.090271 17893 log.cc:824] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:36.090312 17894 log.cc:824] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:36.092133 17895 tablet_bootstrap.cc:654] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:36.092335 17893 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: No bootstrap required, opened a new log
I20260620 04:39:36.092528 17893 ts_tablet_manager.cc:1397] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:39:36.092537 17894 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: No bootstrap required, opened a new log
I20260620 04:39:36.092721 17894 ts_tablet_manager.cc:1397] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:36.093349 17895 log.cc:824] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:36.095266 17895 tablet_bootstrap.cc:492] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: No bootstrap required, opened a new log
I20260620 04:39:36.095443 17895 ts_tablet_manager.cc:1397] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:39:36.095656 17893 raft_consensus.cc:348] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.095896 17893 raft_consensus.cc:374] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:36.095968 17893 raft_consensus.cc:729] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dfe04b97324dde8910c0519edc2b11, State: Initialized, Role: FOLLOWER
I20260620 04:39:36.096422 17893 consensus_queue.cc:260] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.097018 17876 heartbeater.cc:503] Master 127.0.98.62:42923 was elected leader, sending a full tablet report...
I20260620 04:39:36.097035 17894 raft_consensus.cc:348] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.097255 17894 raft_consensus.cc:374] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:36.097301 17894 raft_consensus.cc:729] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e15eb4834f4a5288c8ede0c2c3de20, State: Initialized, Role: FOLLOWER
I20260620 04:39:36.097370 17893 ts_tablet_manager.cc:1428] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:36.097685 17894 consensus_queue.cc:260] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.098567 17894 ts_tablet_manager.cc:1428] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:39:36.098927 17608 heartbeater.cc:503] Master 127.0.98.62:42923 was elected leader, sending a full tablet report...
I20260620 04:39:36.099014 17895 raft_consensus.cc:348] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.099241 17895 raft_consensus.cc:374] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:36.099305 17895 raft_consensus.cc:729] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094ee67b1d2746079a5ca23facbd7559, State: Initialized, Role: FOLLOWER
I20260620 04:39:36.099676 17895 consensus_queue.cc:260] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.100299 17742 heartbeater.cc:503] Master 127.0.98.62:42923 was elected leader, sending a full tablet report...
I20260620 04:39:36.100586 17895 ts_tablet_manager.cc:1428] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 04:39:36.277797 17877 tablet.cc:2369] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:39:36.289278 17743 tablet.cc:2369] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:39:36.319804 17609 tablet.cc:2369] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:39:36.399813 17901 raft_consensus.cc:482] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:36.400032 17901 raft_consensus.cc:504] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.401247 17901 leader_election.cc:288] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1e15eb4834f4a5288c8ede0c2c3de20 (127.0.98.1:45025), 56dfe04b97324dde8910c0519edc2b11 (127.0.98.3:45425)
I20260620 04:39:36.405462 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.401650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53063 (local address 127.0.98.3:45425)
0620 04:39:36.401826 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.401832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.401964 (+ 132us) server_negotiation.cc:407] Connection header received
0620 04:39:36.402096 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.402100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.402153 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.402240 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:36.402581 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.403098 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.404339 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.404889 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.404995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.405062 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.405250 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.405326 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 04:39:36.405473 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.401534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45425 (local address 127.0.98.2:53063)
0620 04:39:36.401856 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.401872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.401998 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.402272 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.402274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.402298 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:36.402447 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.402455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.403224 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.403227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.404208 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.404217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.405035 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.405038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.405113 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.405135 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.405218 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.405313 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260620 04:39:36.405776 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.401655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37723 (local address 127.0.98.1:45025)
0620 04:39:36.401819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.401825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.402117 (+ 292us) server_negotiation.cc:407] Connection header received
0620 04:39:36.402242 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.402245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.402295 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.402366 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:36.402847 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.403492 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.404615 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.405177 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.405305 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.405359 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.405589 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.405665 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:39:36.406090 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.401535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45025 (local address 127.0.98.2:37723)
0620 04:39:36.401995 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.402007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.402116 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:36.402401 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:36.402403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:36.402417 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:36.402541 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.402550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.403641 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.403643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.404492 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:36.404501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.405342 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.405345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:36.405441 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.405461 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:39:36.405908 (+ 447us) client_negotiation.cc:240] Negotiation successful
0620 04:39:36.405992 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":180,"thread_start_us":60,"threads_started":1}
I20260620 04:39:36.406279 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7805c60f85ac45b2b3ef74ddd2fc70e2" candidate_uuid: "094ee67b1d2746079a5ca23facbd7559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dfe04b97324dde8910c0519edc2b11" is_pre_election: true
I20260620 04:39:36.406591 17830 raft_consensus.cc:2454] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094ee67b1d2746079a5ca23facbd7559 in term 0.
I20260620 04:39:36.406653 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7805c60f85ac45b2b3ef74ddd2fc70e2" candidate_uuid: "094ee67b1d2746079a5ca23facbd7559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" is_pre_election: true
I20260620 04:39:36.406898 17562 raft_consensus.cc:2454] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094ee67b1d2746079a5ca23facbd7559 in term 0.
I20260620 04:39:36.407044 17629 leader_election.cc:302] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094ee67b1d2746079a5ca23facbd7559, 56dfe04b97324dde8910c0519edc2b11; no voters:
I20260620 04:39:36.407325 17901 raft_consensus.cc:2781] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:36.407420 17901 raft_consensus.cc:482] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:36.407452 17901 raft_consensus.cc:3037] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:36.408576 17901 raft_consensus.cc:504] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.408958 17901 leader_election.cc:288] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [CANDIDATE]: Term 1 election: Requested vote from peers e1e15eb4834f4a5288c8ede0c2c3de20 (127.0.98.1:45025), 56dfe04b97324dde8910c0519edc2b11 (127.0.98.3:45425)
I20260620 04:39:36.409376 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7805c60f85ac45b2b3ef74ddd2fc70e2" candidate_uuid: "094ee67b1d2746079a5ca23facbd7559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20"
I20260620 04:39:36.409511 17562 raft_consensus.cc:3037] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:36.409533 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7805c60f85ac45b2b3ef74ddd2fc70e2" candidate_uuid: "094ee67b1d2746079a5ca23facbd7559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dfe04b97324dde8910c0519edc2b11"
I20260620 04:39:36.409669 17830 raft_consensus.cc:3037] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:36.410681 17830 raft_consensus.cc:2454] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094ee67b1d2746079a5ca23facbd7559 in term 1.
I20260620 04:39:36.410786 17562 raft_consensus.cc:2454] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094ee67b1d2746079a5ca23facbd7559 in term 1.
I20260620 04:39:36.410961 17629 leader_election.cc:302] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094ee67b1d2746079a5ca23facbd7559, 56dfe04b97324dde8910c0519edc2b11; no voters:
I20260620 04:39:36.411146 17901 raft_consensus.cc:2781] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:36.411626 17901 raft_consensus.cc:686] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 1 LEADER]: Becoming Leader. State: Replica: 094ee67b1d2746079a5ca23facbd7559, State: Running, Role: LEADER
I20260620 04:39:36.411972 17901 consensus_queue.cc:237] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } }
I20260620 04:39:36.415189 17416 catalog_manager.cc:5495] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 094ee67b1d2746079a5ca23facbd7559 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "094ee67b1d2746079a5ca23facbd7559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094ee67b1d2746079a5ca23facbd7559" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 } health_report { overall_health: UNKNOWN } } }
I20260620 04:39:36.439309 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.435554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.0.98.2:34105)
0620 04:39:36.435705 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.435710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.435776 (+ 66us) server_negotiation.cc:407] Connection header received
0620 04:39:36.435856 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.435859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.435909 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.436014 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:36.436313 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.436841 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.437867 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.438095 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.438210 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.438515 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:36.438923 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:36.439055 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.439114 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.439167 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:39:36.445658 17562 raft_consensus.cc:1261] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 1 FOLLOWER]: Refusing update from remote peer 094ee67b1d2746079a5ca23facbd7559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:36.445658 17830 raft_consensus.cc:1261] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 1 FOLLOWER]: Refusing update from remote peer 094ee67b1d2746079a5ca23facbd7559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:36.446411 17901 consensus_queue.cc:1035] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dfe04b97324dde8910c0519edc2b11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:36.446640 17904 consensus_queue.cc:1035] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e15eb4834f4a5288c8ede0c2c3de20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:36.454276 17907 mvcc.cc:205] Tried to move back new op lower bound from 7298786821913640960 to 7298786821785882624. Current Snapshot: MvccSnapshot[applied={T|T < 7298786821913640960}]
I20260620 04:39:36.463186 17417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:36.463379 17417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:36.465586 17417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb5a110a545247c8b081501aa2bd8cc9: Sending DeleteTablet for 3 replicas of tablet 7805c60f85ac45b2b3ef74ddd2fc70e2
I20260620 04:39:36.466427 17810 tablet_service.cc:1514] Processing DeleteTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260620 04:39:36.466459 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38602
I20260620 04:39:36.467082 17917 tablet_replica.cc:330] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: stopping tablet replica
I20260620 04:39:36.467082 17918 tablet_replica.cc:330] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20: stopping tablet replica
I20260620 04:39:36.467422 17917 raft_consensus.cc:2229] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:36.467432 17918 raft_consensus.cc:2229] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:36.467768 17918 raft_consensus.cc:2258] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P e1e15eb4834f4a5288c8ede0c2c3de20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:36.467779 17917 raft_consensus.cc:2258] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:36.468740 17676 tablet_service.cc:1514] Processing DeleteTablet for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51624
I20260620 04:39:36.469198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17479
I20260620 04:39:36.469341 17917 ts_tablet_manager.cc:1905] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:36.472023 17917 ts_tablet_manager.cc:1918] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:36.472105 17917 log.cc:1196] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/7805c60f85ac45b2b3ef74ddd2fc70e2
I20260620 04:39:36.472565 17917 ts_tablet_manager.cc:1939] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 56dfe04b97324dde8910c0519edc2b11: Deleting consensus metadata
I20260620 04:39:36.473937 17401 catalog_manager.cc:4841] TS 56dfe04b97324dde8910c0519edc2b11 (127.0.98.3:45425): tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 (table test-table [id=f5724862ea3148cb904962f73d559e38]) successfully deleted
I20260620 04:39:36.481429 17919 tablet_replica.cc:330] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: stopping tablet replica
I20260620 04:39:36.481878 17919 raft_consensus.cc:2229] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:39:36.482308 17919 raft_consensus.cc:2258] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:36.484790 17919 ts_tablet_manager.cc:1905] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:36.487511 17919 ts_tablet_manager.cc:1918] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:36.487613 17919 log.cc:1196] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/7805c60f85ac45b2b3ef74ddd2fc70e2
I20260620 04:39:36.487924 17919 ts_tablet_manager.cc:1939] T 7805c60f85ac45b2b3ef74ddd2fc70e2 P 094ee67b1d2746079a5ca23facbd7559: Deleting consensus metadata
W20260620 04:39:36.488302 17403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45025 (error 108)
I20260620 04:39:36.489218 17404 catalog_manager.cc:4841] TS 094ee67b1d2746079a5ca23facbd7559 (127.0.98.2:34105): tablet 7805c60f85ac45b2b3ef74ddd2fc70e2 (table test-table [id=f5724862ea3148cb904962f73d559e38]) successfully deleted
I20260620 04:39:36.489454 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.489083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45025 (local address 127.0.0.1:38618)
0620 04:39:36.489271 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.489352 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 04:39:36.489660 17403 catalog_manager.cc:4584] TS e1e15eb4834f4a5288c8ede0c2c3de20 (127.0.98.1:45025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7805c60f85ac45b2b3ef74ddd2fc70e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45025: connect: Connection refused (error 111)
I20260620 04:39:36.490159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17613
I20260620 04:39:36.495797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17747
I20260620 04:39:36.501377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17385
2026-06-20T04:39:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2601 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T04:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:36Z Disabled control of system clock
I20260620 04:39:36.531818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42621
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:36.638412 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:36.638654 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:36.638689 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:36.642652 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:36.642704 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:36.642719 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:36.642735 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:36.642750 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:36.647589 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:36.648653 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:36.649984 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:36.658417 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:36.658445 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:36.658603 17924 server_base.cc:1034] running on GCE node
W20260620 04:39:36.658445 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:36.659301 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:36.660396 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:36.661600 17924 hybrid_clock.cc:648] HybridClock initialized: now 1781930376661582 us; error 29 us; skew 500 ppm
I20260620 04:39:36.663445 17924 webserver.cc:460] Webserver started at http://127.0.98.62:38835/ using document root <none> and password file <none>
I20260620 04:39:36.664013 17924 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:36.664104 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:36.664363 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:36.666440 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "c51f5838275b4becbbe8931ba239bc50"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "dfbb565c6d34599593ae0f6199b704f3"
server_key_iv: "c57c9182d4555a3c009205a6385b9e91"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.667012 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c51f5838275b4becbbe8931ba239bc50"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "dfbb565c6d34599593ae0f6199b704f3"
server_key_iv: "c57c9182d4555a3c009205a6385b9e91"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.670508 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:39:36.672860 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.674093 17924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:36.674335 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "c51f5838275b4becbbe8931ba239bc50"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "dfbb565c6d34599593ae0f6199b704f3"
server_key_iv: "c57c9182d4555a3c009205a6385b9e91"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.674455 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:36.696710 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:36.697395 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:36.697587 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:36.704761 17924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42621
I20260620 04:39:36.704768 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42621 every 8 connection(s)
I20260620 04:39:36.705669 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:36.707365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 17924
I20260620 04:39:36.707504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:36.707808 392 external_mini_cluster.cc:1428] Setting key f5917c76471e73bfb984254bb39d2ed9
I20260620 04:39:36.708959 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:36.714962 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: Bootstrap starting.
I20260620 04:39:36.717070 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.709222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.0.98.62:42621)
0620 04:39:36.709817 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.709833 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.709866 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:39:36.710498 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.710513 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:36.710871 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:36.711130 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:36.711453 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.712468 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.713637 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:36.714143 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:36.714273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:36.715234 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:36.715239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:36.715251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:36.715627 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:36.715686 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:36.715706 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:36.715786 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:36.716072 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 04:39:36.716374 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260620 04:39:36.717413 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:36.718125 17991 log.cc:824] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:36.719913 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: No bootstrap required, opened a new log
I20260620 04:39:36.723449 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } }
I20260620 04:39:36.723670 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:36.723749 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51f5838275b4becbbe8931ba239bc50, State: Initialized, Role: FOLLOWER
I20260620 04:39:36.724221 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } }
I20260620 04:39:36.724382 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:36.724444 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:36.724534 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:36.725360 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } }
I20260620 04:39:36.725684 17991 leader_election.cc:302] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51f5838275b4becbbe8931ba239bc50; no voters:
I20260620 04:39:36.726014 17991 leader_election.cc:288] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:36.726147 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:36.726394 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [term 1 LEADER]: Becoming Leader. State: Replica: c51f5838275b4becbbe8931ba239bc50, State: Running, Role: LEADER
I20260620 04:39:36.726655 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } }
I20260620 04:39:36.727172 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:36.728331 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51f5838275b4becbbe8931ba239bc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } } }
I20260620 04:39:36.728480 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:36.728839 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51f5838275b4becbbe8931ba239bc50. Latest consensus state: current_term: 1 leader_uuid: "c51f5838275b4becbbe8931ba239bc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51f5838275b4becbbe8931ba239bc50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42621 } } }
I20260620 04:39:36.729007 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:36.729432 18005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:36.731863 18005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:36.737805 18005 catalog_manager.cc:1347] Generated new cluster ID: b47324269763434ba6a988b2d037f0ef
I20260620 04:39:36.737895 18005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:36.789325 18005 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:36.790549 18005 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:36.799295 18005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: Generated new TSK 0
I20260620 04:39:36.800143 18005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:36.813459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:36.923651 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:36.923918 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:36.923993 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:36.927991 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:36.928061 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:36.928141 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:36.932861 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:36.933847 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:36.935133 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:36.944789 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:36.944792 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:36.944768 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:36.944842 18015 server_base.cc:1034] running on GCE node
I20260620 04:39:36.945340 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:36.946000 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:36.947212 18015 hybrid_clock.cc:648] HybridClock initialized: now 1781930376947190 us; error 31 us; skew 500 ppm
I20260620 04:39:36.949249 18015 webserver.cc:460] Webserver started at http://127.0.98.1:35761/ using document root <none> and password file <none>
I20260620 04:39:36.949811 18015 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:36.949903 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:36.950124 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:36.952044 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "696450fc055541d0b200eda355bf5964"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "bdf4a52827a1873d0a370045f564a978"
server_key_iv: "77cbf648b3a14236ce633fa08002043f"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.952641 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "696450fc055541d0b200eda355bf5964"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "bdf4a52827a1873d0a370045f564a978"
server_key_iv: "77cbf648b3a14236ce633fa08002043f"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.956089 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:36.958552 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.959836 18015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:36.959980 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "696450fc055541d0b200eda355bf5964"
format_stamp: "Formatted at 2026-06-20 04:39:36 on dist-test-slave-zmf4"
server_key: "bdf4a52827a1873d0a370045f564a978"
server_key_iv: "77cbf648b3a14236ce633fa08002043f"
server_key_version: "encryptionkey@0"
I20260620 04:39:36.960093 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:36.983052 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:36.983693 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:36.983886 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:36.984500 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:36.985383 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:36.985453 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.985519 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:36.985570 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:36.994601 18015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38095
I20260620 04:39:36.994639 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38095 every 8 connection(s)
I20260620 04:39:36.995486 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:36.999890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18015
I20260620 04:39:37.000025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:37.000382 392 external_mini_cluster.cc:1428] Setting key 97de8f020d8bad17201d2a6fdf4e8352
I20260620 04:39:37.002403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:37.005609 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.997341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50131 (local address 127.0.98.62:42621)
0620 04:39:36.997517 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:36.997521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:36.998483 (+ 962us) server_negotiation.cc:407] Connection header received
0620 04:39:36.999982 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:36.999986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.000054 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.000151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:37.001174 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.001797 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.003500 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.003713 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.003852 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.005080 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:37.005083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:37.005085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:37.005097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:37.005115 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:37.005124 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:37.005201 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.005382 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.005461 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 04:39:37.006397 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:36.997677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42621 (local address 127.0.98.1:50131)
0620 04:39:36.998323 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:36.998362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:36.999701 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.000350 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.000358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.000461 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:37.001027 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.001041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.001916 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.001920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.003362 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.003377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.003885 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.003889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.004066 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.004794 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:37.004795 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:37.004836 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:37.004901 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:37.005215 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:37.005226 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:37.005234 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.005368 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.005631 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260620 04:39:37.007639 18143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42621
I20260620 04:39:37.007889 18143 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:37.008436 18143 heartbeater.cc:511] Master 127.0.98.62:42621 requested a full tablet report, sending...
I20260620 04:39:37.010073 17955 ts_manager.cc:194] Registered new tserver with Master: 696450fc055541d0b200eda355bf5964 (127.0.98.1:38095)
I20260620 04:39:37.011866 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50131
W20260620 04:39:37.113826 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:37.114053 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:37.114089 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:37.118006 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:37.118055 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:37.118106 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:37.122653 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:37.123560 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:37.124801 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:37.134416 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:37.134435 18146 server_base.cc:1034] running on GCE node
W20260620 04:39:37.134451 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:37.134467 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:37.135063 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:37.135648 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:37.136857 18146 hybrid_clock.cc:648] HybridClock initialized: now 1781930377136837 us; error 40 us; skew 500 ppm
I20260620 04:39:37.138751 18146 webserver.cc:460] Webserver started at http://127.0.98.2:39465/ using document root <none> and password file <none>
I20260620 04:39:37.139325 18146 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:37.139417 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:37.139626 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:37.141654 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "df04bfd380dd492bb7a78a4fe13221b7"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "dbbbb0d3f4d79e6402e275bfa7cef8ca"
server_key_iv: "ba6bea34c39148b30b49f331d0a4478e"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.142212 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "df04bfd380dd492bb7a78a4fe13221b7"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "dbbbb0d3f4d79e6402e275bfa7cef8ca"
server_key_iv: "ba6bea34c39148b30b49f331d0a4478e"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.145880 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 04:39:37.148459 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.149689 18146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:39:37.149840 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "df04bfd380dd492bb7a78a4fe13221b7"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "dbbbb0d3f4d79e6402e275bfa7cef8ca"
server_key_iv: "ba6bea34c39148b30b49f331d0a4478e"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.149964 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:37.183044 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:37.183730 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:37.183943 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:37.184547 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:37.185480 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:37.185552 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.185624 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:37.185669 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.194335 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43439
I20260620 04:39:37.194361 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43439 every 8 connection(s)
I20260620 04:39:37.195209 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:37.199447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18146
I20260620 04:39:37.199606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:37.199898 392 external_mini_cluster.cc:1428] Setting key f1919af9defdb44e28c85f958de4d2e0
I20260620 04:39:37.202008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:37.205173 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.196942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46341 (local address 127.0.98.62:42621)
0620 04:39:37.197110 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.197115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.198068 (+ 953us) server_negotiation.cc:407] Connection header received
0620 04:39:37.199439 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.199444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.199504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.199600 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:37.200763 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.201371 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.202644 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.203035 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.203159 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.204576 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:37.204583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:37.204589 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:37.204606 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:37.204632 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:37.204646 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:37.204730 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.204885 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.204958 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 04:39:37.205919 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.197279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42621 (local address 127.0.98.2:46341)
0620 04:39:37.197926 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.197969 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.199204 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.199864 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.199877 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.200019 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:37.200605 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.200624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.201515 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.201518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.202520 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.202530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.203209 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.203215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.203453 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.204144 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:37.204148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:37.204246 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:37.204335 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:37.204738 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:37.204746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:37.204750 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.204846 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.205059 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260620 04:39:37.207257 18274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42621
I20260620 04:39:37.207518 18274 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:37.208156 18274 heartbeater.cc:511] Master 127.0.98.62:42621 requested a full tablet report, sending...
I20260620 04:39:37.209385 17955 ts_manager.cc:194] Registered new tserver with Master: df04bfd380dd492bb7a78a4fe13221b7 (127.0.98.2:43439)
I20260620 04:39:37.210039 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46341
W20260620 04:39:37.314050 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:37.314370 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:37.314451 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:37.318491 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:37.318584 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:37.318683 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:37.323808 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:37.324900 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:37.326256 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:37.335783 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:37.335795 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:37.335831 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:37.336098 18277 server_base.cc:1034] running on GCE node
I20260620 04:39:37.336541 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:37.337183 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:37.338415 18277 hybrid_clock.cc:648] HybridClock initialized: now 1781930377338397 us; error 29 us; skew 500 ppm
I20260620 04:39:37.340153 18277 webserver.cc:460] Webserver started at http://127.0.98.3:35887/ using document root <none> and password file <none>
I20260620 04:39:37.340796 18277 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:37.340899 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:37.341123 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:37.343103 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "51fdbe3d705642d2b5b77557f921ed82"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "90f924a492123d4e900b92ca3ab20369"
server_key_iv: "b896695e8a957461f1a26ecc228e01ae"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.343672 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51fdbe3d705642d2b5b77557f921ed82"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "90f924a492123d4e900b92ca3ab20369"
server_key_iv: "b896695e8a957461f1a26ecc228e01ae"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.347281 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:39:37.349773 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.351007 18277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:37.351151 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "51fdbe3d705642d2b5b77557f921ed82"
format_stamp: "Formatted at 2026-06-20 04:39:37 on dist-test-slave-zmf4"
server_key: "90f924a492123d4e900b92ca3ab20369"
server_key_iv: "b896695e8a957461f1a26ecc228e01ae"
server_key_version: "encryptionkey@0"
I20260620 04:39:37.351267 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:37.364099 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:37.364768 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:37.364916 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:37.365540 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:37.366379 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:37.366425 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.366462 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:37.366477 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:37.375164 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41697
I20260620 04:39:37.375190 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41697 every 8 connection(s)
I20260620 04:39:37.375983 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:37.378823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18277
I20260620 04:39:37.378947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:37.379254 392 external_mini_cluster.cc:1428] Setting key bad30e8eb8381764ba21b8e010982943
I20260620 04:39:37.385110 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.377766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35001 (local address 127.0.98.62:42621)
0620 04:39:37.377889 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.377893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.378998 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 04:39:37.380601 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.380604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.380657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.380762 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:37.381610 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.382131 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.383221 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.383450 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.383554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.384620 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:37.384624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:37.384628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:37.384646 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:37.384664 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:37.384671 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:37.384750 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.384906 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.384969 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:39:37.385897 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.378125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42621 (local address 127.0.98.3:35001)
0620 04:39:37.378788 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.378842 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.380409 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.380941 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.380948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.381027 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:37.381473 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.381484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.382251 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.382255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.383098 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.383106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.383598 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.383602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.383757 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.384381 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:37.384383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:37.384418 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:37.384473 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:37.384751 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:37.384762 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:37.384777 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.384904 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.385164 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260620 04:39:37.387009 18405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42621
I20260620 04:39:37.387247 18405 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:37.387729 18405 heartbeater.cc:511] Master 127.0.98.62:42621 requested a full tablet report, sending...
I20260620 04:39:37.388806 17955 ts_manager.cc:194] Registered new tserver with Master: 51fdbe3d705642d2b5b77557f921ed82 (127.0.98.3:41697)
I20260620 04:39:37.389371 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35001
I20260620 04:39:37.393020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:37.398625 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.394875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.98.62:42621)
0620 04:39:37.394992 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.394995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.395227 (+ 232us) server_negotiation.cc:407] Connection header received
0620 04:39:37.395363 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.395367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.395406 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.395495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:37.395759 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.396333 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.397156 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.397387 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.397492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.398214 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:37.398218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:37.398221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:37.398233 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:37.398252 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:37.398268 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:37.398341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.398444 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.398493 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:39:37.401631 17955 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:37.404703 17955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:37.418673 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.413330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43439 (local address 127.0.0.1:33962)
0620 04:39:37.414136 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.414153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.414235 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.414887 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.414890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.414904 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:37.414980 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.414987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.416281 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.416286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.417347 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.417358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.418111 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.418214 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.418249 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.418370 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.418486 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":29,"thread_start_us":81,"threads_started":1}
I20260620 04:39:37.418668 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.413719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.0.98.2:43439)
0620 04:39:37.414478 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.414493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.414518 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:39:37.414588 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.414593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.414755 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.414918 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:37.415070 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.416148 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.417470 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.417989 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.418297 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.418395 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.418485 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":109,"threads_started":1}
I20260620 04:39:37.418725 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.413058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41697 (local address 127.0.0.1:41780)
0620 04:39:37.413593 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.413625 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.413757 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.414528 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.414533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.414554 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:37.414677 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.414683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.415825 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.415828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.417155 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.417166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.418227 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.418390 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.418431 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.418481 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.418557 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"spinlock_wait_cycles":7168,"thread_start_us":101,"threads_started":1}
I20260620 04:39:37.419111 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.412998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38095 (local address 127.0.0.1:55746)
0620 04:39:37.413579 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.413625 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.413759 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.414644 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.414647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.414660 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:37.414734 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.414740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.416169 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.416172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.417395 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.417404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.418697 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.418815 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.418837 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.418920 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.418984 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":50,"spinlock_wait_cycles":3840,"thread_start_us":67,"threads_started":1}
I20260620 04:39:37.419229 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.413119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.0.98.1:38095)
0620 04:39:37.414046 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.414058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.414081 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:39:37.414153 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.414158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.414356 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.414507 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:37.414873 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.416044 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.417524 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.418351 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418469 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.418637 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.418952 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.419048 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":104,"threads_started":1}
I20260620 04:39:37.420161 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.413722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.98.3:41697)
0620 04:39:37.414046 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.414059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.414081 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:39:37.414153 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.414158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.414356 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.414818 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:37.414950 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.415704 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.417296 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.418083 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.418835 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.419012 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.419059 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.419162 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":140,"threads_started":1}
I20260620 04:39:37.422366 18077 tablet_service.cc:1467] Processing CreateTablet for tablet 6600524113b447e1a315954dd2dd2afe (DEFAULT_TABLE table=test-table [id=53308201e93a4ef4a184dcf4cdff08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:37.422608 18339 tablet_service.cc:1467] Processing CreateTablet for tablet 6600524113b447e1a315954dd2dd2afe (DEFAULT_TABLE table=test-table [id=53308201e93a4ef4a184dcf4cdff08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:37.422705 18208 tablet_service.cc:1467] Processing CreateTablet for tablet 6600524113b447e1a315954dd2dd2afe (DEFAULT_TABLE table=test-table [id=53308201e93a4ef4a184dcf4cdff08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:37.423486 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6600524113b447e1a315954dd2dd2afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:37.423497 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6600524113b447e1a315954dd2dd2afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:37.423512 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6600524113b447e1a315954dd2dd2afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:37.430474 18423 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Bootstrap starting.
I20260620 04:39:37.430999 18422 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Bootstrap starting.
I20260620 04:39:37.431912 18424 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Bootstrap starting.
I20260620 04:39:37.432657 18423 tablet_bootstrap.cc:654] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:37.433051 18422 tablet_bootstrap.cc:654] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:37.433399 18423 log.cc:824] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:37.433842 18424 tablet_bootstrap.cc:654] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:37.433816 18422 log.cc:824] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:37.434594 18424 log.cc:824] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:37.435456 18422 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: No bootstrap required, opened a new log
I20260620 04:39:37.435456 18423 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: No bootstrap required, opened a new log
I20260620 04:39:37.435642 18423 ts_tablet_manager.cc:1397] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:37.435642 18422 ts_tablet_manager.cc:1397] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:39:37.436277 18424 tablet_bootstrap.cc:492] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: No bootstrap required, opened a new log
I20260620 04:39:37.436458 18424 ts_tablet_manager.cc:1397] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:37.439333 18424 raft_consensus.cc:348] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.439567 18424 raft_consensus.cc:374] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:37.439618 18424 raft_consensus.cc:729] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696450fc055541d0b200eda355bf5964, State: Initialized, Role: FOLLOWER
I20260620 04:39:37.440050 18424 consensus_queue.cc:260] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.439937 18423 raft_consensus.cc:348] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.439937 18422 raft_consensus.cc:348] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.440212 18423 raft_consensus.cc:374] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:37.440222 18422 raft_consensus.cc:374] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:37.440275 18423 raft_consensus.cc:729] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df04bfd380dd492bb7a78a4fe13221b7, State: Initialized, Role: FOLLOWER
I20260620 04:39:37.440284 18422 raft_consensus.cc:729] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fdbe3d705642d2b5b77557f921ed82, State: Initialized, Role: FOLLOWER
I20260620 04:39:37.440688 18143 heartbeater.cc:503] Master 127.0.98.62:42621 was elected leader, sending a full tablet report...
I20260620 04:39:37.440716 18423 consensus_queue.cc:260] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.440716 18422 consensus_queue.cc:260] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.441202 18424 ts_tablet_manager.cc:1428] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:37.441578 18422 ts_tablet_manager.cc:1428] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:39:37.441578 18423 ts_tablet_manager.cc:1428] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 04:39:37.441885 18274 heartbeater.cc:503] Master 127.0.98.62:42621 was elected leader, sending a full tablet report...
I20260620 04:39:37.442049 18405 heartbeater.cc:503] Master 127.0.98.62:42621 was elected leader, sending a full tablet report...
W20260620 04:39:37.446034 18275 tablet.cc:2369] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:39:37.496639 18144 tablet.cc:2369] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:39:37.554695 18428 raft_consensus.cc:482] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:37.554965 18428 raft_consensus.cc:504] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.556155 18428 leader_election.cc:288] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51fdbe3d705642d2b5b77557f921ed82 (127.0.98.3:41697), df04bfd380dd492bb7a78a4fe13221b7 (127.0.98.2:43439)
I20260620 04:39:37.560242 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.556788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54121 (local address 127.0.98.3:41697)
0620 04:39:37.556932 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.556937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.556951 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:39:37.557126 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.557129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.557167 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.557263 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:37.557551 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.558045 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559208 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.559582 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559701 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.559756 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.560020 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.560093 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:39:37.560232 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.556560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57385 (local address 127.0.98.2:43439)
0620 04:39:37.556690 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.556695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.556888 (+ 193us) server_negotiation.cc:407] Connection header received
0620 04:39:37.557066 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.557071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.557137 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.557193 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:37.557534 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.558053 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559170 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.559576 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559688 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.559753 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.559989 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.560076 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:39:37.560256 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.556465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43439 (local address 127.0.98.1:57385)
0620 04:39:37.556788 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.556800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.556938 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.557234 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.557239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.557258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:37.557414 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.557420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.558186 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.558189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.559047 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.559055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.559702 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.559860 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.559884 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.559944 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.560074 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":48,"thread_start_us":92,"threads_started":1}
I20260620 04:39:37.560261 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.556465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41697 (local address 127.0.98.1:54121)
0620 04:39:37.556888 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.556903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:37.557018 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:37.557298 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:37.557303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:37.557319 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:37.557425 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.557429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.558158 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.558164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.559059 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:37.559068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.559727 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.559730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:37.559873 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.559898 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:39:37.559989 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 04:39:37.560099 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":161,"thread_start_us":70,"threads_started":1}
I20260620 04:39:37.560926 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6600524113b447e1a315954dd2dd2afe" candidate_uuid: "696450fc055541d0b200eda355bf5964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df04bfd380dd492bb7a78a4fe13221b7" is_pre_election: true
I20260620 04:39:37.561048 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6600524113b447e1a315954dd2dd2afe" candidate_uuid: "696450fc055541d0b200eda355bf5964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fdbe3d705642d2b5b77557f921ed82" is_pre_election: true
I20260620 04:39:37.561225 18228 raft_consensus.cc:2454] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696450fc055541d0b200eda355bf5964 in term 0.
I20260620 04:39:37.561340 18359 raft_consensus.cc:2454] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696450fc055541d0b200eda355bf5964 in term 0.
I20260620 04:39:37.561717 18032 leader_election.cc:302] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696450fc055541d0b200eda355bf5964, df04bfd380dd492bb7a78a4fe13221b7; no voters:
I20260620 04:39:37.562026 18428 raft_consensus.cc:2781] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:37.562111 18428 raft_consensus.cc:482] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:37.562179 18428 raft_consensus.cc:3037] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:37.563180 18428 raft_consensus.cc:504] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.563517 18428 leader_election.cc:288] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [CANDIDATE]: Term 1 election: Requested vote from peers 51fdbe3d705642d2b5b77557f921ed82 (127.0.98.3:41697), df04bfd380dd492bb7a78a4fe13221b7 (127.0.98.2:43439)
I20260620 04:39:37.563905 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6600524113b447e1a315954dd2dd2afe" candidate_uuid: "696450fc055541d0b200eda355bf5964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df04bfd380dd492bb7a78a4fe13221b7"
I20260620 04:39:37.564007 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6600524113b447e1a315954dd2dd2afe" candidate_uuid: "696450fc055541d0b200eda355bf5964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fdbe3d705642d2b5b77557f921ed82"
I20260620 04:39:37.564030 18228 raft_consensus.cc:3037] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:37.564167 18359 raft_consensus.cc:3037] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:37.565057 18228 raft_consensus.cc:2454] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696450fc055541d0b200eda355bf5964 in term 1.
I20260620 04:39:37.565140 18359 raft_consensus.cc:2454] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696450fc055541d0b200eda355bf5964 in term 1.
I20260620 04:39:37.565351 18032 leader_election.cc:302] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696450fc055541d0b200eda355bf5964, df04bfd380dd492bb7a78a4fe13221b7; no voters:
I20260620 04:39:37.565505 18428 raft_consensus.cc:2781] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:37.565776 18428 raft_consensus.cc:686] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 1 LEADER]: Becoming Leader. State: Replica: 696450fc055541d0b200eda355bf5964, State: Running, Role: LEADER
I20260620 04:39:37.566082 18428 consensus_queue.cc:237] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } }
I20260620 04:39:37.569226 17950 catalog_manager.cc:5495] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696450fc055541d0b200eda355bf5964 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "696450fc055541d0b200eda355bf5964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696450fc055541d0b200eda355bf5964" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 } health_report { overall_health: UNKNOWN } } }
I20260620 04:39:37.609151 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.605561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.98.1:38095)
0620 04:39:37.605699 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:37.605703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:37.605736 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:39:37.605807 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:37.605810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:37.605880 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:37.605959 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:37.606197 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.606677 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.607542 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:37.607765 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:37.607854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:37.608162 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:37.608583 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:37.608682 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:37.608812 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 04:39:37.608857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:39:37.614809 18228 raft_consensus.cc:1261] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 1 FOLLOWER]: Refusing update from remote peer 696450fc055541d0b200eda355bf5964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:37.614809 18359 raft_consensus.cc:1261] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 1 FOLLOWER]: Refusing update from remote peer 696450fc055541d0b200eda355bf5964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:37.615543 18428 consensus_queue.cc:1035] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fdbe3d705642d2b5b77557f921ed82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:37.615736 18433 consensus_queue.cc:1035] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df04bfd380dd492bb7a78a4fe13221b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:37.622709 18436 mvcc.cc:205] Tried to move back new op lower bound from 7298786826703626240 to 7298786826512883712. Current Snapshot: MvccSnapshot[applied={T|T < 7298786826703626240}]
I20260620 04:39:37.623237 18437 mvcc.cc:205] Tried to move back new op lower bound from 7298786826703626240 to 7298786826512883712. Current Snapshot: MvccSnapshot[applied={T|T < 7298786826703626240}]
I20260620 04:39:37.624783 18438 mvcc.cc:205] Tried to move back new op lower bound from 7298786826703626240 to 7298786826512883712. Current Snapshot: MvccSnapshot[applied={T|T < 7298786826703626240}]
I20260620 04:39:37.631481 17950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:37.631624 17950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:37.634039 17950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c51f5838275b4becbbe8931ba239bc50: Sending DeleteTablet for 3 replicas of tablet 6600524113b447e1a315954dd2dd2afe
I20260620 04:39:37.634824 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet 6600524113b447e1a315954dd2dd2afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:37 UTC) from {username='slave'} at 127.0.0.1:55746
I20260620 04:39:37.634944 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6600524113b447e1a315954dd2dd2afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:37 UTC) from {username='slave'} at 127.0.0.1:41780
I20260620 04:39:37.635341 18446 tablet_replica.cc:330] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964: stopping tablet replica
I20260620 04:39:37.635634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18015
I20260620 04:39:37.635605 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet 6600524113b447e1a315954dd2dd2afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:37 UTC) from {username='slave'} at 127.0.0.1:33962
I20260620 04:39:37.635792 18446 raft_consensus.cc:2229] T 6600524113b447e1a315954dd2dd2afe P 696450fc055541d0b200eda355bf5964 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:39:37.636729 18448 tablet_replica.cc:330] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: stopping tablet replica
I20260620 04:39:37.637048 18448 raft_consensus.cc:2229] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:37.637346 18448 raft_consensus.cc:2258] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:37.637707 18447 tablet_replica.cc:330] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: stopping tablet replica
I20260620 04:39:37.637989 18447 raft_consensus.cc:2229] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:37.638257 18447 raft_consensus.cc:2258] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:37.638935 18448 ts_tablet_manager.cc:1905] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:37.639541 18447 ts_tablet_manager.cc:1905] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:37.641350 18448 ts_tablet_manager.cc:1918] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:37.641460 18448 log.cc:1196] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/6600524113b447e1a315954dd2dd2afe
I20260620 04:39:37.641772 18448 ts_tablet_manager.cc:1939] T 6600524113b447e1a315954dd2dd2afe P df04bfd380dd492bb7a78a4fe13221b7: Deleting consensus metadata
I20260620 04:39:37.642843 17941 catalog_manager.cc:4841] TS df04bfd380dd492bb7a78a4fe13221b7 (127.0.98.2:43439): tablet 6600524113b447e1a315954dd2dd2afe (table test-table [id=53308201e93a4ef4a184dcf4cdff08ad]) successfully deleted
I20260620 04:39:37.643038 18447 ts_tablet_manager.cc:1918] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:37.643141 18447 log.cc:1196] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/6600524113b447e1a315954dd2dd2afe
I20260620 04:39:37.643465 18447 ts_tablet_manager.cc:1939] T 6600524113b447e1a315954dd2dd2afe P 51fdbe3d705642d2b5b77557f921ed82: Deleting consensus metadata
W20260620 04:39:37.644385 17940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38095 (error 108)
I20260620 04:39:37.644618 17942 catalog_manager.cc:4841] TS 51fdbe3d705642d2b5b77557f921ed82 (127.0.98.3:41697): tablet 6600524113b447e1a315954dd2dd2afe (table test-table [id=53308201e93a4ef4a184dcf4cdff08ad]) successfully deleted
I20260620 04:39:37.645205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18146
I20260620 04:39:37.645463 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:37.645156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38095 (local address 127.0.0.1:55764)
0620 04:39:37.645295 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:37.645367 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:39:37.646135 17940 catalog_manager.cc:4584] TS 696450fc055541d0b200eda355bf5964 (127.0.98.1:38095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6600524113b447e1a315954dd2dd2afe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38095: connect: Connection refused (error 111)
I20260620 04:39:37.651386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18277
I20260620 04:39:37.656817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 17924
2026-06-20T04:39:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1154 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:37 dist-test-slave-zmf4 krb5kdc[18452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:37 dist-test-slave-zmf4 krb5kdc[18452](info): set up 2 sockets
Jun 20 04:39:37 dist-test-slave-zmf4 krb5kdc[18452](info): commencing operation
krb5kdc: starting...
W20260620 04:39:38.708067 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:38 dist-test-slave-zmf4 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:39:38.832022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:38.939582 18465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:38.939843 18465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:38.939884 18465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:38.943737 18465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:38.943794 18465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:38.943813 18465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:38.943830 18465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:38.943845 18465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:38.949267 18465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:38.950222 18465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:38.951131 18465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:39:38.959498 18465 server_base.cc:1034] running on GCE node
W20260620 04:39:38.959476 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:38.959470 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:38.959765 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:38.960498 18465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:38.961513 18465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:38.962690 18465 hybrid_clock.cc:648] HybridClock initialized: now 1781930378962668 us; error 34 us; skew 500 ppm
Jun 20 04:39:38 dist-test-slave-zmf4 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:38.965479 18465 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:39:38.966347 18465 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260620 04:39:38.966907 18465 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:38.966997 18465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:38.967223 18465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:38.969274 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "83b5098d03a1473b939e287406db904f"
format_stamp: "Formatted at 2026-06-20 04:39:38 on dist-test-slave-zmf4"
server_key: "cc02efae89d2e0bbb405982a8654b7be"
server_key_iv: "fdc0926d5650c67671bb3c394ecdd78d"
server_key_version: "encryptionkey@0"
I20260620 04:39:38.969831 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "83b5098d03a1473b939e287406db904f"
format_stamp: "Formatted at 2026-06-20 04:39:38 on dist-test-slave-zmf4"
server_key: "cc02efae89d2e0bbb405982a8654b7be"
server_key_iv: "fdc0926d5650c67671bb3c394ecdd78d"
server_key_version: "encryptionkey@0"
I20260620 04:39:38.973615 18465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:39:38.975978 18480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:38.977523 18465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:39:38.977684 18465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "83b5098d03a1473b939e287406db904f"
format_stamp: "Formatted at 2026-06-20 04:39:38 on dist-test-slave-zmf4"
server_key: "cc02efae89d2e0bbb405982a8654b7be"
server_key_iv: "fdc0926d5650c67671bb3c394ecdd78d"
server_key_version: "encryptionkey@0"
I20260620 04:39:38.977809 18465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:39.002143 18465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:39.005318 18465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:39.005548 18465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:39.012347 18465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37449
I20260620 04:39:39.012372 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37449 every 8 connection(s)
I20260620 04:39:39.013255 18465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:39.015877 18533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:39.018002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18465
I20260620 04:39:39.018142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:39.018431 392 external_mini_cluster.cc:1428] Setting key e628c584a3f8ca919e2fb200ac7e9d94
I20260620 04:39:39.022491 18533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: Bootstrap starting.
I20260620 04:39:39.024720 18533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:39.025460 18533 log.cc:824] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:39.027341 18533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: No bootstrap required, opened a new log
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:39.030520 18533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } }
I20260620 04:39:39.030725 18533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:39.030802 18533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b5098d03a1473b939e287406db904f, State: Initialized, Role: FOLLOWER
I20260620 04:39:39.031248 18533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } }
I20260620 04:39:39.031373 18533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:39.031453 18533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:39.031571 18533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:39.032482 18533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } }
I20260620 04:39:39.032804 18533 leader_election.cc:302] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b5098d03a1473b939e287406db904f; no voters:
I20260620 04:39:39.033135 18533 leader_election.cc:288] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:39.033257 18538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:39.033492 18538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [term 1 LEADER]: Becoming Leader. State: Replica: 83b5098d03a1473b939e287406db904f, State: Running, Role: LEADER
I20260620 04:39:39.033836 18538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } }
I20260620 04:39:39.034377 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.019951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.0.98.62:37449)
0620 04:39:39.020682 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.020704 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.020768 (+ 64us) server_negotiation.cc:407] Connection header received
0620 04:39:39.021741 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.021767 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.022329 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.022718 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:39.023531 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.024608 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.025543 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.026053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.026226 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.028982 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:39.028988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:39.029001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:39.029026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:39.031325 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.031877 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.031883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.031889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.032004 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.032321 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.032325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.032327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.032771 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:39.032978 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.033223 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.033530 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":160,"threads_started":1}
I20260620 04:39:39.034375 18533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:39.034978 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b5098d03a1473b939e287406db904f. Latest consensus state: current_term: 1 leader_uuid: "83b5098d03a1473b939e287406db904f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } } }
I20260620 04:39:39.035104 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:39.035703 18539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b5098d03a1473b939e287406db904f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b5098d03a1473b939e287406db904f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37449 } } }
I20260620 04:39:39.035811 18539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:39.036617 18542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:39.039678 18542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:39.046543 18542 catalog_manager.cc:1347] Generated new cluster ID: 4bab577af54c40e8a1ae2b02b5830b3c
I20260620 04:39:39.046629 18542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:39.058979 18542 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:39.060037 18542 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:39.066536 18542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: Generated new TSK 0
I20260620 04:39:39.067272 18542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:39:39.103849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:39.221429 18559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:39.221709 18559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:39.221752 18559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:39.225889 18559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:39.225950 18559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:39.226015 18559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:39.231493 18559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:39.232987 18559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:39.233952 18559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:39.244309 18565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:39.244318 18564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:39.244318 18567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:39.244752 18559 server_base.cc:1034] running on GCE node
I20260620 04:39:39.245162 18559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:39.245820 18559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:39.247018 18559 hybrid_clock.cc:648] HybridClock initialized: now 1781930379246991 us; error 40 us; skew 500 ppm
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:39.249903 18559 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:39:39.250836 18559 webserver.cc:460] Webserver started at http://127.0.98.1:42585/ using document root <none> and password file <none>
I20260620 04:39:39.251400 18559 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:39.251505 18559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:39.251715 18559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:39.253783 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "72379554f7524f5b99ba9310aad335db"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "53bd2260483ba249839889935de5e1ff"
server_key_iv: "65c0370cc0d853504f6e85866b44ac78"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.254338 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72379554f7524f5b99ba9310aad335db"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "53bd2260483ba249839889935de5e1ff"
server_key_iv: "65c0370cc0d853504f6e85866b44ac78"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.258131 18559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:39:39.260655 18574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.262090 18559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:39.262246 18559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "72379554f7524f5b99ba9310aad335db"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "53bd2260483ba249839889935de5e1ff"
server_key_iv: "65c0370cc0d853504f6e85866b44ac78"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.262365 18559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:39.278816 18559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:39.283172 18559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:39.283402 18559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:39.284010 18559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:39.284847 18559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:39.284929 18559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.285006 18559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:39.285050 18559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.294704 18559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43905
I20260620 04:39:39.294721 18687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43905 every 8 connection(s)
I20260620 04:39:39.295590 18559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:39.300868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18559
I20260620 04:39:39.301003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:39.301267 392 external_mini_cluster.cc:1428] Setting key 7997084a62118863a9b2a3b977cfcbd5
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:39:39.311139 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.297582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33251 (local address 127.0.98.62:37449)
0620 04:39:39.297740 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.297745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.299724 (+ 1979us) server_negotiation.cc:407] Connection header received
0620 04:39:39.300669 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.300673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.300736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.300845 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:39.302440 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.303174 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.304376 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.304700 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.304822 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.307512 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:39.307516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:39.307519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:39.307552 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:39.309062 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.309576 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.309580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.309582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.309664 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.310002 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.310007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.310009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.310240 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:39.310356 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.310843 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.310966 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:39:39.311962 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.297991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37449 (local address 127.0.98.1:33251)
0620 04:39:39.299549 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.299589 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.300421 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.301082 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.301091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.301623 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:39.302252 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.302271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.303307 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.303309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.304184 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.304249 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.304864 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.304868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.305044 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.305647 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:39.305663 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:39.307342 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:39.309194 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.309199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.309214 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.309472 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.309763 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.309766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.309768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.309908 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.310354 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:39.310361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:39.310601 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.310783 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.311105 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1383,"thread_start_us":182,"threads_started":1}
I20260620 04:39:39.313604 18688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:39:39.313867 18688 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:39.314374 18688 heartbeater.cc:511] Master 127.0.98.62:37449 requested a full tablet report, sending...
I20260620 04:39:39.316093 18497 ts_manager.cc:194] Registered new tserver with Master: 72379554f7524f5b99ba9310aad335db (127.0.98.1:43905)
I20260620 04:39:39.317786 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:39:39.331802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:39.474166 18693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:39.474472 18693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:39.474567 18693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:39.478706 18693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:39.478806 18693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:39.478896 18693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:39.484635 18693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:39.485708 18693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:39.486716 18693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:39.496639 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:39.496644 18701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:39.496677 18693 server_base.cc:1034] running on GCE node
W20260620 04:39:39.496898 18699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:39.497654 18693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:39.498281 18693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:39.499480 18693 hybrid_clock.cc:648] HybridClock initialized: now 1781930379499433 us; error 62 us; skew 500 ppm
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:39.502329 18693 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:39:39.503208 18693 webserver.cc:460] Webserver started at http://127.0.98.2:35627/ using document root <none> and password file <none>
I20260620 04:39:39.503726 18693 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:39.503793 18693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:39.503957 18693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:39.505990 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "98d0c0c11e184d12ab559ceca2fbaf4e"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "9b59f716d9a64dfad2b5c7efd9fc2141"
server_key_iv: "e740e0dd61d27d8c888dec7889bc581d"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.506491 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98d0c0c11e184d12ab559ceca2fbaf4e"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "9b59f716d9a64dfad2b5c7efd9fc2141"
server_key_iv: "e740e0dd61d27d8c888dec7889bc581d"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.510591 18693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 04:39:39.513927 18708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.515460 18693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:39:39.515579 18693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "98d0c0c11e184d12ab559ceca2fbaf4e"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "9b59f716d9a64dfad2b5c7efd9fc2141"
server_key_iv: "e740e0dd61d27d8c888dec7889bc581d"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.515681 18693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:39.535063 18693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:39.538540 18693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:39.538738 18693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:39.539304 18693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:39.540138 18693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:39.540421 18693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.540494 18693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:39.540552 18693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.549553 18693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44317
I20260620 04:39:39.549605 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44317 every 8 connection(s)
I20260620 04:39:39.550470 18693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:39.551795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18693
I20260620 04:39:39.551913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:39.552182 392 external_mini_cluster.cc:1428] Setting key b173dd3cf38c67d0f89fedc5f3d60b6b
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:39:39.566488 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.552525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37631 (local address 127.0.98.62:37449)
0620 04:39:39.552758 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.552764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.554142 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 04:39:39.555471 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.555473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.555558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.555667 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:39.556950 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.557845 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.559127 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.559584 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.559716 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.562202 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:39.562206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:39.562209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:39.562237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:39.563798 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.564507 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.564510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.564511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.564604 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.565059 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.565062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.565064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.565301 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:39.565416 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.566072 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.566216 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 04:39:39.567370 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.553094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37449 (local address 127.0.98.2:37631)
0620 04:39:39.553958 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.554012 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.555224 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.555862 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.555869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.556288 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:39.556820 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.556833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.557993 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.557996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.558992 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.559001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.559745 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.559748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.559913 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.560570 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:39.560586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:39.562002 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:39.563944 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.563949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.563962 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.564384 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.564723 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.564726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.564727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.564942 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.565428 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:39.565434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:39.565758 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.565970 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.566239 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":126,"threads_started":1}
I20260620 04:39:39.568650 18822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37449
I20260620 04:39:39.568907 18822 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:39.569458 18822 heartbeater.cc:511] Master 127.0.98.62:37449 requested a full tablet report, sending...
I20260620 04:39:39.570570 18497 ts_manager.cc:194] Registered new tserver with Master: 98d0c0c11e184d12ab559ceca2fbaf4e (127.0.98.2:44317)
I20260620 04:39:39.571153 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:39:39.583907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:39.698333 18827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:39.698587 18827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:39.698630 18827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:39.702697 18827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:39.702755 18827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:39.702841 18827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:39.708400 18827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:39.709446 18827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:39.710469 18827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:39.721192 18835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:39.721119 18832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:39.721115 18833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:39.721372 18827 server_base.cc:1034] running on GCE node
I20260620 04:39:39.721801 18827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:39.722445 18827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:39.723641 18827 hybrid_clock.cc:648] HybridClock initialized: now 1781930379723606 us; error 47 us; skew 500 ppm
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:39.726663 18827 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:39:39.727609 18827 webserver.cc:460] Webserver started at http://127.0.98.3:46583/ using document root <none> and password file <none>
I20260620 04:39:39.728169 18827 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:39.728324 18827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:39.728559 18827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:39.730584 18827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "68bce60d098e4f79b0e3bb37029cb925"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "827784b5c70248452d01df897b93f313"
server_key_iv: "3eaac7eb2c12fee6f709759e6841b244"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.731139 18827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68bce60d098e4f79b0e3bb37029cb925"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "827784b5c70248452d01df897b93f313"
server_key_iv: "3eaac7eb2c12fee6f709759e6841b244"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.735082 18827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:39:39.737766 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.739040 18827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:39.739149 18827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "68bce60d098e4f79b0e3bb37029cb925"
format_stamp: "Formatted at 2026-06-20 04:39:39 on dist-test-slave-zmf4"
server_key: "827784b5c70248452d01df897b93f313"
server_key_iv: "3eaac7eb2c12fee6f709759e6841b244"
server_key_version: "encryptionkey@0"
I20260620 04:39:39.739234 18827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:39.758097 18827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:39.761559 18827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:39.761772 18827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:39.762365 18827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:39.763175 18827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:39.763218 18827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.763293 18827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:39.763332 18827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:39.773667 18827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33709
I20260620 04:39:39.773687 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33709 every 8 connection(s)
I20260620 04:39:39.774598 18827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:39.780546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 18827
I20260620 04:39:39.780700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:39.781028 392 external_mini_cluster.cc:1428] Setting key a85dae9fed28626f072bf5a351b9d939
Jun 20 04:39:39 dist-test-slave-zmf4 krb5kdc[18452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:39.788951 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.776835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57713 (local address 127.0.98.62:37449)
0620 04:39:39.776998 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.777003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.777939 (+ 936us) server_negotiation.cc:407] Connection header received
0620 04:39:39.778756 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.778759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.778812 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.778895 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:39.780086 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.780844 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.782212 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.782543 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.782643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.785298 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:39.785302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:39.785303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:39.785329 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:39.786802 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.787348 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.787351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.787352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.787447 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.787843 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.787846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.787847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.788036 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:39.788122 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.788713 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.788831 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:39:39.789763 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.777243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37449 (local address 127.0.98.3:57713)
0620 04:39:39.777774 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.777817 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.778562 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.779090 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.779097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.779461 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:39.779922 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.779935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.780997 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.781000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.782063 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.782074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.782750 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.782755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.782923 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.783516 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:39.783532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:39.785101 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:39.786941 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.786946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.786957 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.787238 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.787568 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:39.787570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:39.787572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:39.787711 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:39.788156 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:39.788161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:39.788480 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.788650 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.788928 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260620 04:39:39.791088 18956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37449
I20260620 04:39:39.791352 18956 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:39.792016 18956 heartbeater.cc:511] Master 127.0.98.62:37449 requested a full tablet report, sending...
I20260620 04:39:39.793371 18497 ts_manager.cc:194] Registered new tserver with Master: 68bce60d098e4f79b0e3bb37029cb925 (127.0.98.3:33709)
I20260620 04:39:39.793995 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57713
I20260620 04:39:39.796147 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:39.806737 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.798326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.0.98.62:37449)
0620 04:39:39.798513 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.798517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.798663 (+ 146us) server_negotiation.cc:407] Connection header received
0620 04:39:39.798825 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.798828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.798878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.798964 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:39.799651 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.800163 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.801029 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.801301 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.801412 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.803139 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:39.803142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:39.803144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:39.803170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:39.804708 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.805248 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.805251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.805252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.805340 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:39.805729 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:39.805732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:39.805733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:39.805934 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:39.806056 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.806419 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.806547 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:39:39.809964 18497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:39.813256 18497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:39.828680 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.823363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43905 (local address 127.0.0.1:42618)
0620 04:39:39.823852 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.823866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.823930 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.824593 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.824598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.824615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:39.824689 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.824695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.826018 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.826021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.827147 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.827154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.828031 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.828034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.828136 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.828325 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.828389 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.828527 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":146,"thread_start_us":102,"threads_started":1}
I20260620 04:39:39.828678 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.823520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.0.98.1:43905)
0620 04:39:39.824168 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.824175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.824197 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:39:39.824307 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.824312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.824477 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.824613 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:39.824808 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.825892 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.827260 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.827905 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.828226 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.828378 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.828431 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.828523 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":108,"threads_started":1}
I20260620 04:39:39.828727 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.822714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.3:33709)
0620 04:39:39.823178 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.823192 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.823927 (+ 735us) server_negotiation.cc:407] Connection header received
0620 04:39:39.824016 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.824021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.824239 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.824393 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:39.824630 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.825406 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.827098 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.827846 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.827953 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.828069 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.828497 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.828580 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
I20260620 04:39:39.828864 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.822363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.2:44317)
0620 04:39:39.823160 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.823173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.823199 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:39:39.823281 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.823286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.823505 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.823663 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:39.824144 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.825344 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.826918 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.827726 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.827834 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.827993 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.828607 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.828697 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":99,"threads_started":1}
I20260620 04:39:39.829591 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.822336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33709 (local address 127.0.0.1:46520)
0620 04:39:39.823786 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.823813 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.823919 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.824381 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.824384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.824402 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:39.824501 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.824509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.825571 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.825574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.826972 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.826983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.827972 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.827977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.828134 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.828277 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.828389 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.829449 (+ 1060us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":125,"thread_start_us":117,"threads_started":1}
I20260620 04:39:39.830442 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.822265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44317 (local address 127.0.0.1:32940)
0620 04:39:39.822655 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.822700 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.822840 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.823832 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.823837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.823857 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:39.824015 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.824022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.825495 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.825502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.826794 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.826804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.828056 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.828060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.828258 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.828318 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.828389 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.828527 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":68,"thread_start_us":90,"threads_started":1}
I20260620 04:39:39.832350 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 (DEFAULT_TABLE table=test-table [id=6a5c34f23944469c8508f9b1ba2e3301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:39.832341 18622 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 (DEFAULT_TABLE table=test-table [id=6a5c34f23944469c8508f9b1ba2e3301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:39.833318 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb388a10fdc4f02af0a1d16912033d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:39.833367 18622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb388a10fdc4f02af0a1d16912033d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:39.833029 18756 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 (DEFAULT_TABLE table=test-table [id=6a5c34f23944469c8508f9b1ba2e3301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:39.833925 18756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb388a10fdc4f02af0a1d16912033d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:39.839368 18973 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Bootstrap starting.
I20260620 04:39:39.841554 18975 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Bootstrap starting.
I20260620 04:39:39.841624 18974 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: Bootstrap starting.
I20260620 04:39:39.841662 18973 tablet_bootstrap.cc:654] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:39.842648 18973 log.cc:824] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:39.843629 18974 tablet_bootstrap.cc:654] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:39.844082 18975 tablet_bootstrap.cc:654] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:39.844565 18973 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: No bootstrap required, opened a new log
I20260620 04:39:39.844650 18974 log.cc:824] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:39.844791 18973 ts_tablet_manager.cc:1397] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:39:39.845002 18975 log.cc:824] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:39.846629 18974 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: No bootstrap required, opened a new log
I20260620 04:39:39.846827 18974 ts_tablet_manager.cc:1397] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:39.848134 18973 raft_consensus.cc:348] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.848479 18973 raft_consensus.cc:374] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:39.848572 18973 raft_consensus.cc:729] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bce60d098e4f79b0e3bb37029cb925, State: Initialized, Role: FOLLOWER
I20260620 04:39:39.849035 18973 consensus_queue.cc:260] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.849788 18956 heartbeater.cc:503] Master 127.0.98.62:37449 was elected leader, sending a full tablet report...
I20260620 04:39:39.849925 18975 tablet_bootstrap.cc:492] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: No bootstrap required, opened a new log
I20260620 04:39:39.849936 18973 ts_tablet_manager.cc:1428] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:39:39.850132 18975 ts_tablet_manager.cc:1397] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 04:39:39.852732 18974 raft_consensus.cc:348] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.853052 18974 raft_consensus.cc:374] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:39.853158 18974 raft_consensus.cc:729] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72379554f7524f5b99ba9310aad335db, State: Initialized, Role: FOLLOWER
I20260620 04:39:39.853639 18974 consensus_queue.cc:260] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.854559 18688 heartbeater.cc:503] Master 127.0.98.62:37449 was elected leader, sending a full tablet report...
I20260620 04:39:39.854714 18974 ts_tablet_manager.cc:1428] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 04:39:39.854825 18975 raft_consensus.cc:348] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.855736 18975 raft_consensus.cc:374] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:39.855835 18975 raft_consensus.cc:729] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d0c0c11e184d12ab559ceca2fbaf4e, State: Initialized, Role: FOLLOWER
I20260620 04:39:39.856392 18975 consensus_queue.cc:260] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.857074 18822 heartbeater.cc:503] Master 127.0.98.62:37449 was elected leader, sending a full tablet report...
I20260620 04:39:39.857412 18975 ts_tablet_manager.cc:1428] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 04:39:39.874472 18979 raft_consensus.cc:482] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:39.874701 18979 raft_consensus.cc:504] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.876113 18979 leader_election.cc:288] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98d0c0c11e184d12ab559ceca2fbaf4e (127.0.98.2:44317), 72379554f7524f5b99ba9310aad335db (127.0.98.1:43905)
I20260620 04:39:39.880837 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.876368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52315 (local address 127.0.98.2:44317)
0620 04:39:39.876496 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.876503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.876518 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:39:39.876802 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.876808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.876867 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.876955 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:39.877257 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.877973 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.879593 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.880252 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.880365 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.880441 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.880614 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.880697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:39:39.880841 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.876267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44317 (local address 127.0.98.3:52315)
0620 04:39:39.876462 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.876477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.876573 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.876968 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.876971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.876990 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:39.877130 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.877165 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.878085 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.878087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.879434 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.879444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.880378 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.880381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.880462 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.880483 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.880585 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.880674 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":9}
I20260620 04:39:39.880978 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.876548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43905 (local address 127.0.98.3:54817)
0620 04:39:39.877002 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.877014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:39.877106 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:39.877381 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:39.877403 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:39.877421 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:39.877541 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.877547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.878480 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.878484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.879656 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:39.879663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.880561 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.880566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:39.880670 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.880689 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:39:39.880759 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:39:39.880812 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":28,"thread_start_us":274,"threads_started":1}
I20260620 04:39:39.881057 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.876711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54817 (local address 127.0.98.1:43905)
0620 04:39:39.876886 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.876891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.877098 (+ 207us) server_negotiation.cc:407] Connection header received
0620 04:39:39.877228 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.877230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.877280 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.877368 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:39.877684 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.878364 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.879774 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.880442 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.880538 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.880622 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.880785 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.880926 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 04:39:39.881773 18776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb388a10fdc4f02af0a1d16912033d9" candidate_uuid: "68bce60d098e4f79b0e3bb37029cb925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" is_pre_election: true
I20260620 04:39:39.881819 18642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb388a10fdc4f02af0a1d16912033d9" candidate_uuid: "68bce60d098e4f79b0e3bb37029cb925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72379554f7524f5b99ba9310aad335db" is_pre_election: true
I20260620 04:39:39.882066 18776 raft_consensus.cc:2454] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68bce60d098e4f79b0e3bb37029cb925 in term 0.
I20260620 04:39:39.882236 18642 raft_consensus.cc:2454] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68bce60d098e4f79b0e3bb37029cb925 in term 0.
I20260620 04:39:39.882599 18845 leader_election.cc:302] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bce60d098e4f79b0e3bb37029cb925, 98d0c0c11e184d12ab559ceca2fbaf4e; no voters:
I20260620 04:39:39.882916 18979 raft_consensus.cc:2781] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:39.883001 18979 raft_consensus.cc:482] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:39.883044 18979 raft_consensus.cc:3037] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:39.884043 18979 raft_consensus.cc:504] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.884457 18979 leader_election.cc:288] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [CANDIDATE]: Term 1 election: Requested vote from peers 98d0c0c11e184d12ab559ceca2fbaf4e (127.0.98.2:44317), 72379554f7524f5b99ba9310aad335db (127.0.98.1:43905)
I20260620 04:39:39.884907 18776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb388a10fdc4f02af0a1d16912033d9" candidate_uuid: "68bce60d098e4f79b0e3bb37029cb925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e"
I20260620 04:39:39.885044 18642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb388a10fdc4f02af0a1d16912033d9" candidate_uuid: "68bce60d098e4f79b0e3bb37029cb925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72379554f7524f5b99ba9310aad335db"
I20260620 04:39:39.885043 18776 raft_consensus.cc:3037] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:39.885202 18642 raft_consensus.cc:3037] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:39.886184 18776 raft_consensus.cc:2454] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68bce60d098e4f79b0e3bb37029cb925 in term 1.
I20260620 04:39:39.886214 18642 raft_consensus.cc:2454] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68bce60d098e4f79b0e3bb37029cb925 in term 1.
I20260620 04:39:39.886492 18845 leader_election.cc:302] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bce60d098e4f79b0e3bb37029cb925, 98d0c0c11e184d12ab559ceca2fbaf4e; no voters:
I20260620 04:39:39.886690 18979 raft_consensus.cc:2781] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:39.886932 18979 raft_consensus.cc:686] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 1 LEADER]: Becoming Leader. State: Replica: 68bce60d098e4f79b0e3bb37029cb925, State: Running, Role: LEADER
I20260620 04:39:39.887269 18979 consensus_queue.cc:237] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } }
I20260620 04:39:39.890702 18496 catalog_manager.cc:5495] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68bce60d098e4f79b0e3bb37029cb925 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "68bce60d098e4f79b0e3bb37029cb925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68bce60d098e4f79b0e3bb37029cb925" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 } health_report { overall_health: UNKNOWN } } }
I20260620 04:39:39.916613 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.912997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.0.98.3:33709)
0620 04:39:39.913160 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:39.913169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:39.913182 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:39:39.913250 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:39.913252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:39.913306 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:39.913417 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:39.913686 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.914195 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.915038 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:39.915256 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:39.915366 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:39.915680 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:39.916104 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:39.916319 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:39.916405 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 04:39:39.916448 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:39:39.922751 18642 raft_consensus.cc:1261] T 4bb388a10fdc4f02af0a1d16912033d9 P 72379554f7524f5b99ba9310aad335db [term 1 FOLLOWER]: Refusing update from remote peer 68bce60d098e4f79b0e3bb37029cb925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:39.922751 18776 raft_consensus.cc:1261] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 1 FOLLOWER]: Refusing update from remote peer 68bce60d098e4f79b0e3bb37029cb925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:39.923604 18979 consensus_queue.cc:1035] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d0c0c11e184d12ab559ceca2fbaf4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:39.923849 18983 consensus_queue.cc:1035] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72379554f7524f5b99ba9310aad335db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:39.932687 18987 mvcc.cc:205] Tried to move back new op lower bound from 7298786836155662336 to 7298786836020797440. Current Snapshot: MvccSnapshot[applied={T|T < 7298786836155662336}]
I20260620 04:39:39.942759 18496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:39.942978 18496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:39.945604 18496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83b5098d03a1473b939e287406db904f: Sending DeleteTablet for 3 replicas of tablet 4bb388a10fdc4f02af0a1d16912033d9
I20260620 04:39:39.946440 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46520
I20260620 04:39:39.946628 18756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32940
I20260620 04:39:39.946960 18996 tablet_replica.cc:330] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: stopping tablet replica
I20260620 04:39:39.947139 18622 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb388a10fdc4f02af0a1d16912033d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42618
I20260620 04:39:39.947295 18996 raft_consensus.cc:2229] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:39:39.947379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18559
I20260620 04:39:39.947721 18996 raft_consensus.cc:2258] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:39.947927 18997 tablet_replica.cc:330] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: stopping tablet replica
I20260620 04:39:39.948333 18997 raft_consensus.cc:2229] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:39.948697 18997 raft_consensus.cc:2258] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:39.949838 18996 ts_tablet_manager.cc:1905] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:39.950259 18997 ts_tablet_manager.cc:1905] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:39.953219 18997 ts_tablet_manager.cc:1918] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:39.953404 18997 log.cc:1196] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/4bb388a10fdc4f02af0a1d16912033d9
I20260620 04:39:39.953385 18996 ts_tablet_manager.cc:1918] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:39.953478 18996 log.cc:1196] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/4bb388a10fdc4f02af0a1d16912033d9
I20260620 04:39:39.953812 18996 ts_tablet_manager.cc:1939] T 4bb388a10fdc4f02af0a1d16912033d9 P 68bce60d098e4f79b0e3bb37029cb925: Deleting consensus metadata
I20260620 04:39:39.953953 18997 ts_tablet_manager.cc:1939] T 4bb388a10fdc4f02af0a1d16912033d9 P 98d0c0c11e184d12ab559ceca2fbaf4e: Deleting consensus metadata
I20260620 04:39:39.955135 18484 catalog_manager.cc:4841] TS 68bce60d098e4f79b0e3bb37029cb925 (127.0.98.3:33709): tablet 4bb388a10fdc4f02af0a1d16912033d9 (table test-table [id=6a5c34f23944469c8508f9b1ba2e3301]) successfully deleted
I20260620 04:39:39.955209 18483 catalog_manager.cc:4841] TS 98d0c0c11e184d12ab559ceca2fbaf4e (127.0.98.2:44317): tablet 4bb388a10fdc4f02af0a1d16912033d9 (table test-table [id=6a5c34f23944469c8508f9b1ba2e3301]) successfully deleted
W20260620 04:39:39.957402 18481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43905 (error 108)
I20260620 04:39:39.958182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18693
I20260620 04:39:39.958493 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:39.958188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43905 (local address 127.0.0.1:42626)
0620 04:39:39.958319 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:39.958396 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 04:39:39.958858 18481 catalog_manager.cc:4584] TS 72379554f7524f5b99ba9310aad335db (127.0.98.1:43905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bb388a10fdc4f02af0a1d16912033d9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43905: connect: Connection refused (error 111)
I20260620 04:39:39.964951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18827
I20260620 04:39:39.969957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 18465
2026-06-20T04:39:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2314 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T04:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:39Z Disabled control of system clock
I20260620 04:39:40.001423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38707
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:40.113068 19003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:40.113379 19003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:40.113420 19003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:40.117604 19003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:40.117666 19003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:40.117682 19003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:40.117699 19003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:40.117715 19003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:40.122587 19003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:40.123714 19003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:40.125234 19003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:40.133617 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.133734 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.133721 19011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.134101 19003 server_base.cc:1034] running on GCE node
I20260620 04:39:40.134850 19003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:40.136338 19003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:40.137907 19003 hybrid_clock.cc:648] HybridClock initialized: now 1781930380137876 us; error 43 us; skew 500 ppm
I20260620 04:39:40.139808 19003 webserver.cc:460] Webserver started at http://127.0.98.62:38155/ using document root <none> and password file <none>
I20260620 04:39:40.140475 19003 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:40.140564 19003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:40.140793 19003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:40.142817 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "2cf60b2351ff41c0affa394b75c1d7c7"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aba8ad84c02d334947338c2a583c36e"
server_key_iv: "a57a7c3350d70ac2fa3750f846dc01af"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.143378 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cf60b2351ff41c0affa394b75c1d7c7"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aba8ad84c02d334947338c2a583c36e"
server_key_iv: "a57a7c3350d70ac2fa3750f846dc01af"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.147064 19003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:39:40.149699 19017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.151013 19003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:40.151162 19003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "2cf60b2351ff41c0affa394b75c1d7c7"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aba8ad84c02d334947338c2a583c36e"
server_key_iv: "a57a7c3350d70ac2fa3750f846dc01af"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.151279 19003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:40.162575 19003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:40.163232 19003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:40.163424 19003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:40.170907 19003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38707
I20260620 04:39:40.170935 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38707 every 8 connection(s)
I20260620 04:39:40.171844 19003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:40.174778 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:40.177677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19003
I20260620 04:39:40.177872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:40.178242 392 external_mini_cluster.cc:1428] Setting key 7090a0f26628f91ebe5912e88fa9e944
I20260620 04:39:40.180477 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: Bootstrap starting.
I20260620 04:39:40.182524 19070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:40.183230 19070 log.cc:824] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:40.185281 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: No bootstrap required, opened a new log
I20260620 04:39:40.187847 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.62:38707)
0620 04:39:40.180669 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.180684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.180718 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:39:40.181323 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.181338 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.181708 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.182031 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:40.182448 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.183520 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.184644 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.185088 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.185205 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.186038 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:40.186043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:40.186055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:40.186410 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:40.186459 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:40.186487 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:40.186561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.186731 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.187018 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260620 04:39:40.188544 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } }
I20260620 04:39:40.188745 19070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:40.188800 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf60b2351ff41c0affa394b75c1d7c7, State: Initialized, Role: FOLLOWER
I20260620 04:39:40.189227 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } }
I20260620 04:39:40.189388 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:40.189459 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:40.189545 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:40.190397 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } }
I20260620 04:39:40.190663 19070 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf60b2351ff41c0affa394b75c1d7c7; no voters:
I20260620 04:39:40.190891 19070 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:40.191028 19075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:40.191293 19075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf60b2351ff41c0affa394b75c1d7c7, State: Running, Role: LEADER
I20260620 04:39:40.191591 19075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } }
I20260620 04:39:40.192061 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:40.193228 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf60b2351ff41c0affa394b75c1d7c7. Latest consensus state: current_term: 1 leader_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } } }
I20260620 04:39:40.193351 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:40.194032 19084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:40.193671 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf60b2351ff41c0affa394b75c1d7c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38707 } } }
I20260620 04:39:40.194496 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:40.196692 19084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:40.202720 19084 catalog_manager.cc:1347] Generated new cluster ID: 2ed1a58c129e4424a9bf11184e1c180f
I20260620 04:39:40.202808 19084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:40.224352 19084 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:40.225240 19084 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:40.231828 19084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: Generated new TSK 0
I20260620 04:39:40.232549 19084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:40.243252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:40.361694 19094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:40.361953 19094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:40.361994 19094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:40.366217 19094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:40.366288 19094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:40.366406 19094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:40.371500 19094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:40.372556 19094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:40.373842 19094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:40.383284 19102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.383420 19094 server_base.cc:1034] running on GCE node
W20260620 04:39:40.383303 19100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.383395 19099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.384039 19094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:40.384676 19094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:40.385907 19094 hybrid_clock.cc:648] HybridClock initialized: now 1781930380385894 us; error 36 us; skew 500 ppm
I20260620 04:39:40.393136 19094 webserver.cc:460] Webserver started at http://127.0.98.1:40637/ using document root <none> and password file <none>
I20260620 04:39:40.393791 19094 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:40.393857 19094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:40.394074 19094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:40.396096 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7c6d17667fc4e77830e8536ea1068e6"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "1b5e343e82200e8e2a4f636ba12f9b0f"
server_key_iv: "2617dc8cb86f274ad6c769e62939876e"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.396981 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c6d17667fc4e77830e8536ea1068e6"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "1b5e343e82200e8e2a4f636ba12f9b0f"
server_key_iv: "2617dc8cb86f274ad6c769e62939876e"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.400916 19094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:39:40.403432 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.404688 19094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:39:40.404811 19094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "d7c6d17667fc4e77830e8536ea1068e6"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "1b5e343e82200e8e2a4f636ba12f9b0f"
server_key_iv: "2617dc8cb86f274ad6c769e62939876e"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.404927 19094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:40.424718 19094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:40.425377 19094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:40.425571 19094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:40.426133 19094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:40.427034 19094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:40.427109 19094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.427181 19094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:40.427225 19094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.436647 19094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37519
I20260620 04:39:40.436671 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37519 every 8 connection(s)
I20260620 04:39:40.437836 19094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:40.440093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19094
I20260620 04:39:40.440256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:40.440563 392 external_mini_cluster.cc:1428] Setting key 31741e14a80a24a4006549418b05b125
I20260620 04:39:40.442636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:40.449041 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.439790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58145 (local address 127.0.98.62:38707)
0620 04:39:40.440006 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.440013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.441479 (+ 1466us) server_negotiation.cc:407] Connection header received
0620 04:39:40.443272 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.443275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.443317 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.443390 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:40.444514 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.445048 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.446867 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.447096 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.447184 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.448534 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:40.448537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:40.448539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:40.448550 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:40.448569 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:40.448577 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:40.448640 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.448868 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.448922 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:39:40.450235 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.440505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38707 (local address 127.0.98.1:58145)
0620 04:39:40.441339 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.441389 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.443045 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.443555 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.443566 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.443671 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:40.444376 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.444392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.445183 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.445189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.446704 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.446718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.447274 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.447281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.447516 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.448176 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:40.448181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:40.448278 (+ 97us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:40.448369 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:40.448674 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:40.448685 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:40.448695 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.448828 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.449158 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":219,"threads_started":1}
I20260620 04:39:40.451558 19222 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38707
I20260620 04:39:40.451792 19222 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:40.452354 19222 heartbeater.cc:511] Master 127.0.98.62:38707 requested a full tablet report, sending...
I20260620 04:39:40.453931 19034 ts_manager.cc:194] Registered new tserver with Master: d7c6d17667fc4e77830e8536ea1068e6 (127.0.98.1:37519)
I20260620 04:39:40.455125 19034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58145
W20260620 04:39:40.558296 19225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:40.558547 19225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:40.558583 19225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:40.562784 19225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:40.562847 19225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:40.562940 19225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:40.567699 19225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:40.568769 19225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:40.570075 19225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:40.579499 19231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.579522 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.579509 19230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.579813 19225 server_base.cc:1034] running on GCE node
I20260620 04:39:40.580240 19225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:40.580891 19225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:40.582069 19225 hybrid_clock.cc:648] HybridClock initialized: now 1781930380582030 us; error 51 us; skew 500 ppm
I20260620 04:39:40.583969 19225 webserver.cc:460] Webserver started at http://127.0.98.2:34453/ using document root <none> and password file <none>
I20260620 04:39:40.584585 19225 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:40.584647 19225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:40.584848 19225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:40.586813 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aa47023c2f6891af1612247006b2341"
server_key_iv: "ef29b713d4c1871bd3cca6ccb79e503c"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.587371 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aa47023c2f6891af1612247006b2341"
server_key_iv: "ef29b713d4c1871bd3cca6ccb79e503c"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.591073 19225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:39:40.593655 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.594854 19225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:40.594977 19225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "5aa47023c2f6891af1612247006b2341"
server_key_iv: "ef29b713d4c1871bd3cca6ccb79e503c"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.595085 19225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:40.606246 19225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:40.606873 19225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:40.607066 19225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:40.607673 19225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:40.608645 19225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:40.608717 19225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.608799 19225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:40.608837 19225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.617511 19225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39719
I20260620 04:39:40.617595 19352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39719 every 8 connection(s)
I20260620 04:39:40.618440 19225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:40.619786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19225
I20260620 04:39:40.619917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:40.620289 392 external_mini_cluster.cc:1428] Setting key 708e5a09e8dca330db4b086d2a41096b
I20260620 04:39:40.622575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:40.631042 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.620869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41325 (local address 127.0.98.62:38707)
0620 04:39:40.621014 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.621018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.622909 (+ 1891us) server_negotiation.cc:407] Connection header received
0620 04:39:40.624449 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.624453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.624517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.624623 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:40.625708 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.626458 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.628452 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.628692 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.628810 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.630468 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:40.630475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:40.630478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:40.630496 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:40.630515 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:40.630527 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:40.630600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.630854 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.630917 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:39:40.632460 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.621537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38707 (local address 127.0.98.2:41325)
0620 04:39:40.622730 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.622798 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.624155 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.624829 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.624836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.624930 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:40.625436 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.625449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.626621 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.626628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.628292 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.628307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.628895 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.628902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.629158 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.629915 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:40.629919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:40.629985 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:40.630090 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:40.630635 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:40.630648 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:40.630656 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.630822 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.631198 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":275,"threads_started":1}
I20260620 04:39:40.633814 19353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38707
I20260620 04:39:40.634064 19353 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:40.634611 19353 heartbeater.cc:511] Master 127.0.98.62:38707 requested a full tablet report, sending...
I20260620 04:39:40.635721 19034 ts_manager.cc:194] Registered new tserver with Master: ae35fcb8d3cf47a89cb2720fcdc77b79 (127.0.98.2:39719)
I20260620 04:39:40.636391 19034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41325
W20260620 04:39:40.739130 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:40.739383 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:40.739424 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:40.743472 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:40.743531 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:40.743587 19356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:40.748407 19356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:40.749522 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:40.750864 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:40.760938 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:40.760962 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.761008 19356 server_base.cc:1034] running on GCE node
W20260620 04:39:40.760962 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:40.761667 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:40.762264 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:40.763460 19356 hybrid_clock.cc:648] HybridClock initialized: now 1781930380763426 us; error 46 us; skew 500 ppm
I20260620 04:39:40.765589 19356 webserver.cc:460] Webserver started at http://127.0.98.3:37159/ using document root <none> and password file <none>
I20260620 04:39:40.766196 19356 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:40.766259 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:40.766471 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:40.769172 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "9045869b644c467bbc4216a009cfd9b1"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "3e2e932e241810d3d71357dfa5eb05c6"
server_key_iv: "cd9beb52457f042a6528a99bb7d4383d"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.769905 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9045869b644c467bbc4216a009cfd9b1"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "3e2e932e241810d3d71357dfa5eb05c6"
server_key_iv: "cd9beb52457f042a6528a99bb7d4383d"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.774370 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:39:40.777374 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.778707 19356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:40.778851 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "9045869b644c467bbc4216a009cfd9b1"
format_stamp: "Formatted at 2026-06-20 04:39:40 on dist-test-slave-zmf4"
server_key: "3e2e932e241810d3d71357dfa5eb05c6"
server_key_iv: "cd9beb52457f042a6528a99bb7d4383d"
server_key_version: "encryptionkey@0"
I20260620 04:39:40.778970 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:40.796916 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:40.797806 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:40.798031 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:40.798678 19356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:40.799937 19356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:40.800011 19356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.800083 19356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:40.800127 19356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:40.809366 19483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34529 every 8 connection(s)
I20260620 04:39:40.809356 19356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34529
I20260620 04:39:40.810473 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:40.819963 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.812522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54973 (local address 127.0.98.62:38707)
0620 04:39:40.812679 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.812684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.813796 (+ 1112us) server_negotiation.cc:407] Connection header received
0620 04:39:40.815267 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.815270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.815317 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.815411 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:40.816334 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.816871 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.818052 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.818269 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.818363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.819440 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:40.819443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:40.819445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:40.819457 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:40.819475 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:40.819484 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:40.819566 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.819780 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.819831 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:39:40.820451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19356
I20260620 04:39:40.820570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:40.820987 392 external_mini_cluster.cc:1428] Setting key 1404b9040e323af9fd397df58fc12fec
I20260620 04:39:40.821007 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.812912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38707 (local address 127.0.98.3:54973)
0620 04:39:40.813644 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.813687 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.815050 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.815569 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.815576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.815660 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:40.816160 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.816173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.817014 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.817017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.817915 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.817924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.818440 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.818443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.818579 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.819177 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:40.819179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:40.819217 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:40.819277 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:40.819620 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:40.819628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:40.819632 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.819755 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.820050 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20260620 04:39:40.822461 19484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38707
I20260620 04:39:40.822690 19484 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:40.823199 19484 heartbeater.cc:511] Master 127.0.98.62:38707 requested a full tablet report, sending...
I20260620 04:39:40.824301 19033 ts_manager.cc:194] Registered new tserver with Master: 9045869b644c467bbc4216a009cfd9b1 (127.0.98.3:34529)
I20260620 04:39:40.824858 19033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54973
I20260620 04:39:40.835361 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:40.841650 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.837566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.0.98.62:38707)
0620 04:39:40.837688 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.837691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.837822 (+ 131us) server_negotiation.cc:407] Connection header received
0620 04:39:40.837888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.837894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.837940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.838015 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:40.838406 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.838948 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.839799 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.840006 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.840088 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.840998 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:40.841001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:40.841003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:40.841014 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:40.841032 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:40.841041 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:40.841150 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.841474 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.841527 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:39:40.844962 19033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:40.847740 19033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:40.862392 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.856853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.98.3:34529)
0620 04:39:40.857282 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.857293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.857508 (+ 215us) server_negotiation.cc:407] Connection header received
0620 04:39:40.857715 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.857720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.857917 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.858036 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:40.858599 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.859334 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.860880 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.861597 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.861713 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.861919 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.862119 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.862215 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":141,"threads_started":1}
I20260620 04:39:40.862417 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.856530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37519 (local address 127.0.0.1:60314)
0620 04:39:40.857320 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.857372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.857515 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.858229 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.858232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.858277 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:40.858510 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.858515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.859954 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.859957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.861018 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.861025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.862002 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.862006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.862102 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.862124 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.862172 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.862243 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":25,"spinlock_wait_cycles":12928,"thread_start_us":129,"threads_started":1}
I20260620 04:39:40.862392 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.856342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34529 (local address 127.0.0.1:48350)
0620 04:39:40.857402 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.857429 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.857569 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.858225 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.858230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.858279 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:40.858476 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.858482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.859472 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.859475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.860756 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.860766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.861730 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.861734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.861843 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.861905 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.862088 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.862213 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":768,"spinlock_wait_cycles":28800,"thread_start_us":85,"threads_started":1}
I20260620 04:39:40.863462 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.856342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39719 (local address 127.0.0.1:44252)
0620 04:39:40.857320 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.857372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.857516 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.858229 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.858232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.858284 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:40.858506 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.858514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.861111 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.861113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.861900 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.861908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.863075 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.863079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.863162 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.863183 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.863263 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.863324 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":133,"spinlock_wait_cycles":58240,"thread_start_us":77,"threads_started":1}
I20260620 04:39:40.863713 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.856478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.2:39719)
0620 04:39:40.856822 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.856834 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.857652 (+ 818us) server_negotiation.cc:407] Connection header received
0620 04:39:40.857730 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.857735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.857935 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.858076 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:40.858641 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.859829 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.862006 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.862772 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.862871 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.863012 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.863308 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.863401 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
I20260620 04:39:40.863792 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.857096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.0.98.1:37519)
0620 04:39:40.857811 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.857822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.857842 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:39:40.857907 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.857923 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.858085 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.858225 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:40.858641 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.859813 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.861125 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.861864 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.862834 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.863012 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.863063 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.863148 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":102,"threads_started":1}
I20260620 04:39:40.865423 19418 tablet_service.cc:1467] Processing CreateTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 (DEFAULT_TABLE table=test-table [id=444bb659755c4dcca186d40d9125274b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:40.866022 19156 tablet_service.cc:1467] Processing CreateTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 (DEFAULT_TABLE table=test-table [id=444bb659755c4dcca186d40d9125274b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:40.866338 19418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:40.866829 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:40.869983 19287 tablet_service.cc:1467] Processing CreateTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 (DEFAULT_TABLE table=test-table [id=444bb659755c4dcca186d40d9125274b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:40.870934 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:40.873775 19502 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Bootstrap starting.
I20260620 04:39:40.873844 19501 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Bootstrap starting.
I20260620 04:39:40.875579 19501 tablet_bootstrap.cc:654] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:40.875839 19502 tablet_bootstrap.cc:654] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:40.876407 19501 log.cc:824] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:40.876672 19502 log.cc:824] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:40.877960 19505 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: Bootstrap starting.
I20260620 04:39:40.879292 19501 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: No bootstrap required, opened a new log
I20260620 04:39:40.879421 19502 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: No bootstrap required, opened a new log
I20260620 04:39:40.879530 19501 ts_tablet_manager.cc:1397] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 04:39:40.879601 19502 ts_tablet_manager.cc:1397] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:39:40.880108 19505 tablet_bootstrap.cc:654] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:40.880914 19505 log.cc:824] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:40.882834 19505 tablet_bootstrap.cc:492] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: No bootstrap required, opened a new log
I20260620 04:39:40.883016 19505 ts_tablet_manager.cc:1397] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:40.883162 19501 raft_consensus.cc:348] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.883428 19501 raft_consensus.cc:374] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:40.883481 19501 raft_consensus.cc:729] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c6d17667fc4e77830e8536ea1068e6, State: Initialized, Role: FOLLOWER
I20260620 04:39:40.883803 19502 raft_consensus.cc:348] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.883940 19501 consensus_queue.cc:260] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.884048 19502 raft_consensus.cc:374] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:40.884106 19502 raft_consensus.cc:729] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9045869b644c467bbc4216a009cfd9b1, State: Initialized, Role: FOLLOWER
I20260620 04:39:40.884539 19222 heartbeater.cc:503] Master 127.0.98.62:38707 was elected leader, sending a full tablet report...
I20260620 04:39:40.884552 19502 consensus_queue.cc:260] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.884884 19501 ts_tablet_manager.cc:1428] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260620 04:39:40.885427 19502 ts_tablet_manager.cc:1428] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:39:40.885753 19484 heartbeater.cc:503] Master 127.0.98.62:38707 was elected leader, sending a full tablet report...
I20260620 04:39:40.887171 19505 raft_consensus.cc:348] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.887408 19505 raft_consensus.cc:374] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:40.887493 19505 raft_consensus.cc:729] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae35fcb8d3cf47a89cb2720fcdc77b79, State: Initialized, Role: FOLLOWER
I20260620 04:39:40.887907 19505 consensus_queue.cc:260] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.888806 19505 ts_tablet_manager.cc:1428] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 04:39:40.888984 19353 heartbeater.cc:503] Master 127.0.98.62:38707 was elected leader, sending a full tablet report...
I20260620 04:39:40.921869 19508 raft_consensus.cc:482] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:40.922050 19508 raft_consensus.cc:504] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.923105 19508 leader_election.cc:288] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae35fcb8d3cf47a89cb2720fcdc77b79 (127.0.98.2:39719), d7c6d17667fc4e77830e8536ea1068e6 (127.0.98.1:37519)
I20260620 04:39:40.926644 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.923374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37519 (local address 127.0.98.3:57363)
0620 04:39:40.923495 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.923514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.923599 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.923883 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.923886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.923903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:40.924020 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.924026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.924815 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.924822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.925588 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.925596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.926226 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.926230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.926338 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.926356 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.926444 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.926516 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 04:39:40.926644 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.923494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57363 (local address 127.0.98.1:37519)
0620 04:39:40.923621 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.923625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.923642 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:39:40.923697 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.923703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.923757 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.923869 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:40.924133 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.924699 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.925714 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.926103 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.926221 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.926279 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.926460 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.926523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:39:40.926910 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.923586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56771 (local address 127.0.98.2:39719)
0620 04:39:40.923726 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.923731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.923941 (+ 210us) server_negotiation.cc:407] Connection header received
0620 04:39:40.924085 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.924088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.924135 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.924247 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:40.924509 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.924991 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.926000 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.926377 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.926494 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.926548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.926740 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.926802 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:39:40.926944 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.923374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39719 (local address 127.0.98.3:56771)
0620 04:39:40.923844 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.923854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:40.923967 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:40.924262 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:40.924265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:40.924281 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:40.924395 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.924400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.925105 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.925110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.925865 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:40.925873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.926505 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.926508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:40.926618 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.926635 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:39:40.926723 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 04:39:40.926786 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":47,"thread_start_us":125,"threads_started":1}
I20260620 04:39:40.927374 19176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cf4e9ceaa249db97e0cfc03eef3cd2" candidate_uuid: "9045869b644c467bbc4216a009cfd9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c6d17667fc4e77830e8536ea1068e6" is_pre_election: true
I20260620 04:39:40.927433 19307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cf4e9ceaa249db97e0cfc03eef3cd2" candidate_uuid: "9045869b644c467bbc4216a009cfd9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" is_pre_election: true
I20260620 04:39:40.927686 19176 raft_consensus.cc:2454] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9045869b644c467bbc4216a009cfd9b1 in term 0.
I20260620 04:39:40.927685 19307 raft_consensus.cc:2454] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9045869b644c467bbc4216a009cfd9b1 in term 0.
I20260620 04:39:40.928211 19373 leader_election.cc:302] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9045869b644c467bbc4216a009cfd9b1, ae35fcb8d3cf47a89cb2720fcdc77b79, d7c6d17667fc4e77830e8536ea1068e6; no voters:
I20260620 04:39:40.928488 19508 raft_consensus.cc:2781] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:40.928561 19508 raft_consensus.cc:482] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:40.928612 19508 raft_consensus.cc:3037] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:40.929589 19508 raft_consensus.cc:504] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.929949 19508 leader_election.cc:288] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [CANDIDATE]: Term 1 election: Requested vote from peers ae35fcb8d3cf47a89cb2720fcdc77b79 (127.0.98.2:39719), d7c6d17667fc4e77830e8536ea1068e6 (127.0.98.1:37519)
I20260620 04:39:40.930275 19307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cf4e9ceaa249db97e0cfc03eef3cd2" candidate_uuid: "9045869b644c467bbc4216a009cfd9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79"
I20260620 04:39:40.930320 19176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cf4e9ceaa249db97e0cfc03eef3cd2" candidate_uuid: "9045869b644c467bbc4216a009cfd9b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c6d17667fc4e77830e8536ea1068e6"
I20260620 04:39:40.930390 19307 raft_consensus.cc:3037] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:40.930492 19176 raft_consensus.cc:3037] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:40.931525 19176 raft_consensus.cc:2454] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9045869b644c467bbc4216a009cfd9b1 in term 1.
I20260620 04:39:40.931559 19307 raft_consensus.cc:2454] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9045869b644c467bbc4216a009cfd9b1 in term 1.
I20260620 04:39:40.931859 19371 leader_election.cc:302] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9045869b644c467bbc4216a009cfd9b1, d7c6d17667fc4e77830e8536ea1068e6; no voters:
I20260620 04:39:40.932044 19508 raft_consensus.cc:2781] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:40.932312 19508 raft_consensus.cc:686] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9045869b644c467bbc4216a009cfd9b1, State: Running, Role: LEADER
I20260620 04:39:40.932632 19508 consensus_queue.cc:237] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } }
I20260620 04:39:40.936162 19034 catalog_manager.cc:5495] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9045869b644c467bbc4216a009cfd9b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9045869b644c467bbc4216a009cfd9b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9045869b644c467bbc4216a009cfd9b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 } health_report { overall_health: UNKNOWN } } }
W20260620 04:39:40.938969 19223 tablet.cc:2369] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:39:40.962052 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.958254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.3:34529)
0620 04:39:40.958401 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:40.958407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:40.958423 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:39:40.958494 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:40.958497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:40.958558 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:40.958669 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:40.958919 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.959426 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.960302 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:40.960537 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:40.960645 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:40.960972 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:40.961532 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:40.961729 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:40.961839 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:39:40.961886 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:39:40.967696 19176 raft_consensus.cc:1261] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 1 FOLLOWER]: Refusing update from remote peer 9045869b644c467bbc4216a009cfd9b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:40.967696 19307 raft_consensus.cc:1261] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P ae35fcb8d3cf47a89cb2720fcdc77b79 [term 1 FOLLOWER]: Refusing update from remote peer 9045869b644c467bbc4216a009cfd9b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:40.968406 19511 consensus_queue.cc:1035] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae35fcb8d3cf47a89cb2720fcdc77b79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:40.968554 19508 consensus_queue.cc:1035] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c6d17667fc4e77830e8536ea1068e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:40.975884 19516 mvcc.cc:205] Tried to move back new op lower bound from 7298786840437170176 to 7298786840302747648. Current Snapshot: MvccSnapshot[applied={T|T < 7298786840437170176}]
I20260620 04:39:40.984989 19034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:40.985160 19034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:40.987380 19034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf60b2351ff41c0affa394b75c1d7c7: Sending DeleteTablet for 3 replicas of tablet 33cf4e9ceaa249db97e0cfc03eef3cd2
I20260620 04:39:40.988113 19418 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:40 UTC) from {username='slave'} at 127.0.0.1:48350
I20260620 04:39:40.988276 19156 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:40 UTC) from {username='slave'} at 127.0.0.1:60314
I20260620 04:39:40.988298 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:40 UTC) from {username='slave'} at 127.0.0.1:44252
I20260620 04:39:40.988627 19524 tablet_replica.cc:330] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: stopping tablet replica
I20260620 04:39:40.988668 19525 tablet_replica.cc:330] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6: stopping tablet replica
I20260620 04:39:40.988936 19525 raft_consensus.cc:2229] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:40.988933 19524 raft_consensus.cc:2229] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:39:40.989275 19525 raft_consensus.cc:2258] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P d7c6d17667fc4e77830e8536ea1068e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:40.989318 19524 raft_consensus.cc:2258] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:40.989435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19094
I20260620 04:39:40.991463 19524 ts_tablet_manager.cc:1905] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:40.995270 19524 ts_tablet_manager.cc:1918] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:40.995458 19524 log.cc:1196] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/33cf4e9ceaa249db97e0cfc03eef3cd2
I20260620 04:39:40.995797 19524 ts_tablet_manager.cc:1939] T 33cf4e9ceaa249db97e0cfc03eef3cd2 P 9045869b644c467bbc4216a009cfd9b1: Deleting consensus metadata
I20260620 04:39:40.997042 19021 catalog_manager.cc:4841] TS 9045869b644c467bbc4216a009cfd9b1 (127.0.98.3:34529): tablet 33cf4e9ceaa249db97e0cfc03eef3cd2 (table test-table [id=444bb659755c4dcca186d40d9125274b]) successfully deleted
W20260620 04:39:40.997462 19018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37519 (error 108)
I20260620 04:39:40.998461 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:40.998144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37519 (local address 127.0.0.1:60326)
0620 04:39:40.998294 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:40.998364 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 04:39:40.998519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19225
W20260620 04:39:40.998617 19018 catalog_manager.cc:4584] TS d7c6d17667fc4e77830e8536ea1068e6 (127.0.98.1:37519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33cf4e9ceaa249db97e0cfc03eef3cd2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37519: connect: Connection refused (error 111)
I20260620 04:39:41.004904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19356
I20260620 04:39:41.005106 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:41.004825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39719 (local address 127.0.0.1:44256)
0620 04:39:41.004958 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:41.005014 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 04:39:41.009883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19003
2026-06-20T04:39:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:41 dist-test-slave-zmf4 krb5kdc[19530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:41 dist-test-slave-zmf4 krb5kdc[19530](info): set up 2 sockets
Jun 20 04:39:41 dist-test-slave-zmf4 krb5kdc[19530](info): commencing operation
krb5kdc: starting...
W20260620 04:39:42.072108 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:39:42.195037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:42.303448 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:42.303712 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:42.303753 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:42.307904 19543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:42.307966 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:42.307986 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:42.308003 19543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:42.308018 19543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:42.313547 19543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:42.314497 19543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:42.315411 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:42.323553 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:42.323546 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.323616 19543 server_base.cc:1034] running on GCE node
W20260620 04:39:42.323546 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.324415 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:42.325373 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:42.326570 19543 hybrid_clock.cc:648] HybridClock initialized: now 1781930382326542 us; error 40 us; skew 500 ppm
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:42.329452 19543 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:39:42.330318 19543 webserver.cc:460] Webserver started at http://127.0.98.62:38333/ using document root <none> and password file <none>
I20260620 04:39:42.330901 19543 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:42.330989 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:42.331194 19543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:42.333379 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "e3c50fcb7cd346a692cca5fa8c303458"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "b7e88e3ed680a39870882ed6d8fa6072"
server_key_iv: "0f1485d9eabb5fb9e536930a7be0c832"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.333973 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3c50fcb7cd346a692cca5fa8c303458"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "b7e88e3ed680a39870882ed6d8fa6072"
server_key_iv: "0f1485d9eabb5fb9e536930a7be0c832"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.337519 19543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:39:42.339915 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.341264 19543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:39:42.341418 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "e3c50fcb7cd346a692cca5fa8c303458"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "b7e88e3ed680a39870882ed6d8fa6072"
server_key_iv: "0f1485d9eabb5fb9e536930a7be0c832"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.341540 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:42.363675 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:42.366647 19543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:42.366858 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:42.374143 19543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39075
I20260620 04:39:42.374166 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39075 every 8 connection(s)
I20260620 04:39:42.375087 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:42.377635 19611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:42.380985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19543
I20260620 04:39:42.381093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:42.381405 392 external_mini_cluster.cc:1428] Setting key 9dc2a414fcaa89b25aa204fcf2d04a58
I20260620 04:39:42.383369 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: Bootstrap starting.
I20260620 04:39:42.386010 19611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:42.386838 19611 log.cc:824] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:42.389000 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: No bootstrap required, opened a new log
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:42.393218 19611 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } }
I20260620 04:39:42.393508 19611 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:42.393573 19611 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c50fcb7cd346a692cca5fa8c303458, State: Initialized, Role: FOLLOWER
I20260620 04:39:42.394129 19611 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } }
I20260620 04:39:42.394304 19611 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:42.394393 19611 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:42.394526 19611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:42.395426 19611 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } }
I20260620 04:39:42.395817 19611 leader_election.cc:302] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c50fcb7cd346a692cca5fa8c303458; no voters:
I20260620 04:39:42.396221 19611 leader_election.cc:288] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:42.396298 19616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:42.396471 19616 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [term 1 LEADER]: Becoming Leader. State: Replica: e3c50fcb7cd346a692cca5fa8c303458, State: Running, Role: LEADER
I20260620 04:39:42.396759 19616 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } }
I20260620 04:39:42.396818 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:42.382743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.0.98.62:39075)
0620 04:39:42.383158 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:42.383179 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:42.383225 (+ 46us) server_negotiation.cc:407] Connection header received
0620 04:39:42.384036 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:42.384058 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:42.384494 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:42.384848 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:42.385511 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.386995 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.388063 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.388752 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.388915 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:42.391412 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:42.391421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:42.391444 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:42.391480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:42.393680 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.394167 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.394173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.394179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.394318 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.394699 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.394706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.394708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.395104 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:42.395323 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:42.395564 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:39:42.395846 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260620 04:39:42.397454 19611 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:42.397891 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c50fcb7cd346a692cca5fa8c303458. Latest consensus state: current_term: 1 leader_uuid: "e3c50fcb7cd346a692cca5fa8c303458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } } }
I20260620 04:39:42.398002 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:42.398332 19617 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c50fcb7cd346a692cca5fa8c303458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c50fcb7cd346a692cca5fa8c303458" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39075 } } }
I20260620 04:39:42.398449 19617 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458 [sys.catalog]: This master's current role is: LEADER
W20260620 04:39:42.402132 19631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:39:42.402220 19631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:39:42.402287 19620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:42.402940 19620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:42.408092 19620 catalog_manager.cc:1347] Generated new cluster ID: f1d46bf2b5f8478780c72d81ad2b00e0
I20260620 04:39:42.408156 19620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:42.415177 19620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:42.416358 19620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:42.429482 19620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: Generated new TSK 0
I20260620 04:39:42.430213 19620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:39:42.461659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:42.572243 19637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:42.572500 19637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:42.572541 19637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:42.576607 19637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:42.576655 19637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:42.576713 19637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:42.582793 19637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:42.583884 19637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:42.584946 19637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:42.594228 19643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:42.594285 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.594329 19637 server_base.cc:1034] running on GCE node
W20260620 04:39:42.594285 19645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.594839 19637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:42.595456 19637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:42.596664 19637 hybrid_clock.cc:648] HybridClock initialized: now 1781930382596642 us; error 30 us; skew 500 ppm
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:42.599571 19637 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:39:42.600507 19637 webserver.cc:460] Webserver started at http://127.0.98.1:45495/ using document root <none> and password file <none>
I20260620 04:39:42.601012 19637 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:42.601068 19637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:42.601223 19637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:42.603125 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "944bf06d10f64d98bc40b754a12fe9ff"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "cb5d37f879d062fe5525f186ba005f67"
server_key_iv: "1e6e01310a588c35b3a70448450b4042"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.603623 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "944bf06d10f64d98bc40b754a12fe9ff"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "cb5d37f879d062fe5525f186ba005f67"
server_key_iv: "1e6e01310a588c35b3a70448450b4042"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.607489 19637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:39:42.609966 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.611233 19637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:39:42.611331 19637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "944bf06d10f64d98bc40b754a12fe9ff"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "cb5d37f879d062fe5525f186ba005f67"
server_key_iv: "1e6e01310a588c35b3a70448450b4042"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.611409 19637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:42.630965 19637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:42.634313 19637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:42.634470 19637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:42.634999 19637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:42.635769 19637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:42.635815 19637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.635854 19637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:42.635869 19637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.645020 19637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43203
I20260620 04:39:42.645037 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43203 every 8 connection(s)
I20260620 04:39:42.645965 19637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:42.647720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19637
I20260620 04:39:42.647866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:42.648177 392 external_mini_cluster.cc:1428] Setting key e1771dd253fa48d47f0fdbac902a754d
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:39:42.660715 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:42.647738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56441 (local address 127.0.98.62:39075)
0620 04:39:42.647894 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:42.647900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:42.649026 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 04:39:42.650141 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:42.650145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:42.650202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:42.650307 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:42.651494 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.652127 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.653432 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.653670 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.653781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:42.656603 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:42.656609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:42.656614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:42.656649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:42.658306 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.658854 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.658857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.658858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.658939 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.659331 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.659334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.659336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.659530 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:42.659621 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:42.660291 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 04:39:42.660455 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:39:42.661589 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:42.648269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39075 (local address 127.0.98.1:56441)
0620 04:39:42.648850 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:42.648894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:42.649913 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:42.650486 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:42.650492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:42.650862 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:42.651334 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:42.651347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.652413 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.652417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:42.653293 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:42.653303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.653822 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.653826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:42.653996 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:42.654663 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:42.654680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:42.656429 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:42.658421 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:42.658426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:42.658439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:42.658772 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:42.659053 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:42.659055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:42.659057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:42.659207 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:42.659658 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:42.659672 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:42.659973 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 04:39:42.660138 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 04:39:42.660538 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":141,"threads_started":1}
I20260620 04:39:42.663102 19766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39075
I20260620 04:39:42.663357 19766 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:42.663854 19766 heartbeater.cc:511] Master 127.0.98.62:39075 requested a full tablet report, sending...
I20260620 04:39:42.665495 19575 ts_manager.cc:194] Registered new tserver with Master: 944bf06d10f64d98bc40b754a12fe9ff (127.0.98.1:43203)
I20260620 04:39:42.667212 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:39:42.680055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:42.791860 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:42.792119 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:42.792227 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:42.796397 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:42.796486 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:42.796571 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:42.802029 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:42.803028 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:42.804136 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:42.813690 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:42.813699 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.813982 19771 server_base.cc:1034] running on GCE node
W20260620 04:39:42.813711 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:42.814527 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:42.815176 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:42.816376 19771 hybrid_clock.cc:648] HybridClock initialized: now 1781930382816355 us; error 35 us; skew 500 ppm
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:42.819114 19771 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:39:42.819918 19771 webserver.cc:460] Webserver started at http://127.0.98.2:35659/ using document root <none> and password file <none>
I20260620 04:39:42.820482 19771 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:42.820544 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:42.820699 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:42.822602 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "900c8cb782ac49b4b701e5ca24fc7c2a"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "3d6a66203fab105aeb621f39db3babdb"
server_key_iv: "23ce2d901405e46933f0fb579becdeae"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.823107 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "900c8cb782ac49b4b701e5ca24fc7c2a"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "3d6a66203fab105aeb621f39db3babdb"
server_key_iv: "23ce2d901405e46933f0fb579becdeae"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.826716 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:39:42.829286 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.830485 19771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:39:42.830598 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "900c8cb782ac49b4b701e5ca24fc7c2a"
format_stamp: "Formatted at 2026-06-20 04:39:42 on dist-test-slave-zmf4"
server_key: "3d6a66203fab105aeb621f39db3babdb"
server_key_iv: "23ce2d901405e46933f0fb579becdeae"
server_key_version: "encryptionkey@0"
I20260620 04:39:42.830683 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:42.848090 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:42.851079 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:42.851231 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:42.851745 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:42.852614 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:42.852660 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.852698 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:42.852720 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:42.862394 19771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37125
I20260620 04:39:42.862421 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37125 every 8 connection(s)
I20260620 04:39:42.863268 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:42.866401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19771
I20260620 04:39:42.866536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:42.866843 392 external_mini_cluster.cc:1428] Setting key 17404c0a15813a70c1483513f11181f1
Jun 20 04:39:42 dist-test-slave-zmf4 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:39:42.878175 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:42.865141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33153 (local address 127.0.98.62:39075)
0620 04:39:42.865330 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:42.865333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:42.866437 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 04:39:42.867501 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:42.867506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:42.867561 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:42.867632 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:42.869100 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.869618 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.870911 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.871247 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.871345 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:42.874412 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:42.874416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:42.874420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:42.874451 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:42.875986 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.876580 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.876585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.876587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.876667 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:42.877058 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:42.877061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:42.877063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:42.877245 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:42.877347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:42.877882 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 04:39:42.878002 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 04:39:42.879015 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:42.865481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39075 (local address 127.0.98.2:33153)
0620 04:39:42.866225 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:42.866281 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:42.867289 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:42.867816 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:42.867822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:42.868288 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:42.868952 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:42.868974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.869745 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.869752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:42.870769 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:42.870778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:42.871434 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:42.871441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:42.871678 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:42.872419 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:42.872446 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:42.874240 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:42.876126 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:42.876131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:42.876147 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:42.876471 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:42.876779 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:42.876782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:42.876786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:42.876955 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:42.877357 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:42.877366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:42.877605 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 04:39:42.877790 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 04:39:42.878116 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:39:42.880260 19900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39075
I20260620 04:39:42.880515 19900 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:42.880985 19900 heartbeater.cc:511] Master 127.0.98.62:39075 requested a full tablet report, sending...
I20260620 04:39:42.882020 19575 ts_manager.cc:194] Registered new tserver with Master: 900c8cb782ac49b4b701e5ca24fc7c2a (127.0.98.2:37125)
I20260620 04:39:42.882570 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:39:42.896843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:43.008077 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:43.008359 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:43.008447 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:43.012804 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:43.012871 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:43.012961 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:43.018734 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:43.019820 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:43.020888 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:43.030288 19910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.030308 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.030298 19911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:43.030570 19905 server_base.cc:1034] running on GCE node
I20260620 04:39:43.031036 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:43.031654 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:43.032860 19905 hybrid_clock.cc:648] HybridClock initialized: now 1781930383032845 us; error 27 us; skew 500 ppm
Jun 20 04:39:43 dist-test-slave-zmf4 krb5kdc[19530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:43.036110 19905 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:39:43.037173 19905 webserver.cc:460] Webserver started at http://127.0.98.3:33961/ using document root <none> and password file <none>
I20260620 04:39:43.037776 19905 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:43.037870 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:43.038185 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:43.040176 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "616163951943488284966a9bad264edf"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "ff5f411fb7afaf9e0718dca6ac6072c0"
server_key_iv: "5fc459ad9b3d87f8549f33096fffae2a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.040796 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "616163951943488284966a9bad264edf"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "ff5f411fb7afaf9e0718dca6ac6072c0"
server_key_iv: "5fc459ad9b3d87f8549f33096fffae2a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.044400 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:39:43.046849 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.048136 19905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:43.048307 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "616163951943488284966a9bad264edf"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "ff5f411fb7afaf9e0718dca6ac6072c0"
server_key_iv: "5fc459ad9b3d87f8549f33096fffae2a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.048421 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:43.085405 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:43.088966 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:43.089192 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:43.089933 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:43.090782 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:43.090857 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.090927 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:43.090970 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.101256 19905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34683
I20260620 04:39:43.101446 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34683 every 8 connection(s)
I20260620 04:39:43.102178 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:43.103529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 19905
I20260620 04:39:43.103672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:43.103993 392 external_mini_cluster.cc:1428] Setting key d5756b359d8585b42d32f68c864a58ea
Jun 20 04:39:43 dist-test-slave-zmf4 krb5kdc[19530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:39:43.116303 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.104143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35883 (local address 127.0.98.62:39075)
0620 04:39:43.104322 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.104333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.105232 (+ 899us) server_negotiation.cc:407] Connection header received
0620 04:39:43.106421 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.106424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.106479 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.106553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:43.107682 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.108321 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.109462 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.109691 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.109802 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.112300 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:43.112303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:43.112305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:43.112329 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:43.113879 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:43.114480 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:43.114483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:43.114484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:43.114590 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:43.114997 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:43.115000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:43.115001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:43.115271 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:43.115359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.115993 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.116109 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:39:43.117353 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.104598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39075 (local address 127.0.98.3:35883)
0620 04:39:43.105082 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.105136 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.106202 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.106737 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.106746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.107094 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:43.107538 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.107550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.108444 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.108447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.109303 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.109313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.109864 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.109867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.110031 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.110606 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:39:43.110622 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:43.112080 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:43.114004 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:43.114009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:43.114020 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:43.114342 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:43.114702 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:39:43.114705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:39:43.114706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:39:43.114864 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:39:43.115419 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:43.115425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:43.115734 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.115932 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.116267 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260620 04:39:43.118501 20034 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39075
I20260620 04:39:43.118744 20034 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:43.119292 20034 heartbeater.cc:511] Master 127.0.98.62:39075 requested a full tablet report, sending...
I20260620 04:39:43.120428 19575 ts_manager.cc:194] Registered new tserver with Master: 616163951943488284966a9bad264edf (127.0.98.3:34683)
I20260620 04:39:43.121028 19575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35883
I20260620 04:39:43.130218 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:43.140278 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.132289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.98.62:39075)
0620 04:39:43.132434 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.132438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.132578 (+ 140us) server_negotiation.cc:407] Connection header received
0620 04:39:43.132660 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.132662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.132700 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.132765 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:43.133564 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.134123 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.135091 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.135301 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.135407 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.136945 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:43.136948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:43.136950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:43.136972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:43.138393 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:43.138865 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:43.138868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:43.138869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:43.138948 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:43.139224 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:43.139228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:43.139229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:43.139476 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:43.139608 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.139963 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.140124 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:39:43.143136 19575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:43.146248 19575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:43.161513 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.155551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34683 (local address 127.0.0.1:53918)
0620 04:39:43.156615 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.156630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.156735 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.157179 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.157182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.157199 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:43.157272 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.157291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158465 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.158471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.159937 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.159947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.161041 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.161045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.161144 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161182 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.161231 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.161358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":145,"thread_start_us":100,"threads_started":1}
I20260620 04:39:43.161578 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.155031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.0.98.1:43203)
0620 04:39:43.156300 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.156335 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.156365 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:39:43.156444 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.156449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.156668 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.157155 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:43.157308 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158461 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.160183 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.160789 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.160915 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161061 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.161221 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.161321 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":129,"threads_started":1}
I20260620 04:39:43.161586 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.154853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43203 (local address 127.0.0.1:34918)
0620 04:39:43.155391 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.155444 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.155566 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.156864 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.156870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.156894 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:43.157013 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.157018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158601 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.158604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.160048 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.160060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.160914 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.160917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.161018 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161055 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.161192 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.161341 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":19,"thread_start_us":67,"threads_started":1}
I20260620 04:39:43.162148 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.155920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.98.3:34683)
0620 04:39:43.156614 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.156626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.156719 (+ 93us) server_negotiation.cc:407] Connection header received
0620 04:39:43.156862 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.156870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.157038 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.157191 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:43.157375 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158189 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.160075 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.160898 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.161716 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161868 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.161914 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.162003 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":418,"threads_started":1}
I20260620 04:39:43.162387 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.155548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.0.98.2:37125)
0620 04:39:43.156101 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.156116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.156674 (+ 558us) server_negotiation.cc:407] Connection header received
0620 04:39:43.156929 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.156934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.157077 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.157203 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:43.157622 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158771 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.160314 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.161089 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.161221 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161372 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.162108 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.162211 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":102,"threads_started":1}
I20260620 04:39:43.162717 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.155025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.0.1:55034)
0620 04:39:43.156568 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.156582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.156761 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.157373 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.157377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.157395 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:43.157492 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.157498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.158910 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.158913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.160203 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.160226 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.161817 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.161823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.161949 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.161973 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.162524 (+ 551us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.162599 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"mutex_wait_us":3,"thread_start_us":136,"threads_started":1}
I20260620 04:39:43.164721 19700 tablet_service.cc:1467] Processing CreateTablet for tablet 458954881ec84dd1b593820f142d6c70 (DEFAULT_TABLE table=test-table [id=e94e63f1793c4edfaff03b72ff1a83a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:43.164719 19968 tablet_service.cc:1467] Processing CreateTablet for tablet 458954881ec84dd1b593820f142d6c70 (DEFAULT_TABLE table=test-table [id=e94e63f1793c4edfaff03b72ff1a83a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:43.165663 19700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458954881ec84dd1b593820f142d6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:43.165663 19968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458954881ec84dd1b593820f142d6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:43.170182 19834 tablet_service.cc:1467] Processing CreateTablet for tablet 458954881ec84dd1b593820f142d6c70 (DEFAULT_TABLE table=test-table [id=e94e63f1793c4edfaff03b72ff1a83a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:43.171241 19834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458954881ec84dd1b593820f142d6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:43.171641 20051 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: Bootstrap starting.
I20260620 04:39:43.173267 20052 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Bootstrap starting.
I20260620 04:39:43.174110 20051 tablet_bootstrap.cc:654] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:43.174960 20051 log.cc:824] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:43.175364 20052 tablet_bootstrap.cc:654] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:43.176682 20052 log.cc:824] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:43.177253 20051 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: No bootstrap required, opened a new log
I20260620 04:39:43.177577 20051 ts_tablet_manager.cc:1397] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:39:43.178507 20055 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Bootstrap starting.
I20260620 04:39:43.178634 20052 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: No bootstrap required, opened a new log
I20260620 04:39:43.178992 20052 ts_tablet_manager.cc:1397] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:39:43.180938 20055 tablet_bootstrap.cc:654] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:43.181588 20055 log.cc:824] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:43.182542 20051 raft_consensus.cc:348] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.182811 20051 raft_consensus.cc:374] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:43.182866 20051 raft_consensus.cc:729] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944bf06d10f64d98bc40b754a12fe9ff, State: Initialized, Role: FOLLOWER
I20260620 04:39:43.182786 20052 raft_consensus.cc:348] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.183041 20052 raft_consensus.cc:374] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:43.183105 20052 raft_consensus.cc:729] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616163951943488284966a9bad264edf, State: Initialized, Role: FOLLOWER
I20260620 04:39:43.183265 20055 tablet_bootstrap.cc:492] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: No bootstrap required, opened a new log
I20260620 04:39:43.183347 20051 consensus_queue.cc:260] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.183460 20055 ts_tablet_manager.cc:1397] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:39:43.183457 20052 consensus_queue.cc:260] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.184026 19766 heartbeater.cc:503] Master 127.0.98.62:39075 was elected leader, sending a full tablet report...
I20260620 04:39:43.184391 20052 ts_tablet_manager.cc:1428] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 04:39:43.184594 20034 heartbeater.cc:503] Master 127.0.98.62:39075 was elected leader, sending a full tablet report...
I20260620 04:39:43.185371 20051 ts_tablet_manager.cc:1428] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 04:39:43.186925 20055 raft_consensus.cc:348] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.187198 20055 raft_consensus.cc:374] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:43.187264 20055 raft_consensus.cc:729] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c8cb782ac49b4b701e5ca24fc7c2a, State: Initialized, Role: FOLLOWER
I20260620 04:39:43.187673 20055 consensus_queue.cc:260] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.188302 19900 heartbeater.cc:503] Master 127.0.98.62:39075 was elected leader, sending a full tablet report...
I20260620 04:39:43.188616 20055 ts_tablet_manager.cc:1428] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:39:43.203524 20057 raft_consensus.cc:482] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:43.203711 20057 raft_consensus.cc:504] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.204789 20057 leader_election.cc:288] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900c8cb782ac49b4b701e5ca24fc7c2a (127.0.98.2:37125), 616163951943488284966a9bad264edf (127.0.98.3:34683)
I20260620 04:39:43.208870 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.205135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60371 (local address 127.0.98.3:34683)
0620 04:39:43.205284 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.205289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.205584 (+ 295us) server_negotiation.cc:407] Connection header received
0620 04:39:43.205796 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.205802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.205859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.205938 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:43.206264 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.206765 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.207848 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.208284 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.208396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.208456 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.208671 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.208733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:39:43.208870 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.205250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40369 (local address 127.0.98.2:37125)
0620 04:39:43.205360 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.205364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.205556 (+ 192us) server_negotiation.cc:407] Connection header received
0620 04:39:43.205696 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.205700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.205755 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.205852 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:43.206171 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.206778 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.207834 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.208303 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.208416 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.208477 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.208672 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.208736 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:39:43.208956 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.205062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.98.1:40369)
0620 04:39:43.205453 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.205469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.205585 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.205887 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.205889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.205907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:43.206051 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.206059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.206893 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.206897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.207717 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.207725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.208429 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.208433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.208557 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.208574 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.208658 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.208736 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":6,"thread_start_us":105,"threads_started":1}
I20260620 04:39:43.209125 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.205062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34683 (local address 127.0.98.1:60371)
0620 04:39:43.205507 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.205522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.205642 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.205992 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.205996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.206018 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:43.206145 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.206152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.206875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.206879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.207712 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.207721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.208400 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.208404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.208498 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.208557 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.208643 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.208727 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":55,"thread_start_us":96,"threads_started":1}
I20260620 04:39:43.209611 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458954881ec84dd1b593820f142d6c70" candidate_uuid: "944bf06d10f64d98bc40b754a12fe9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" is_pre_election: true
I20260620 04:39:43.209618 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458954881ec84dd1b593820f142d6c70" candidate_uuid: "944bf06d10f64d98bc40b754a12fe9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616163951943488284966a9bad264edf" is_pre_election: true
I20260620 04:39:43.209887 19854 raft_consensus.cc:2454] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944bf06d10f64d98bc40b754a12fe9ff in term 0.
I20260620 04:39:43.209887 19988 raft_consensus.cc:2454] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944bf06d10f64d98bc40b754a12fe9ff in term 0.
I20260620 04:39:43.210377 19656 leader_election.cc:302] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c8cb782ac49b4b701e5ca24fc7c2a, 944bf06d10f64d98bc40b754a12fe9ff; no voters:
I20260620 04:39:43.210626 20057 raft_consensus.cc:2781] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:43.210706 20057 raft_consensus.cc:482] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:43.210749 20057 raft_consensus.cc:3037] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:43.211711 20057 raft_consensus.cc:504] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.212031 20057 leader_election.cc:288] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [CANDIDATE]: Term 1 election: Requested vote from peers 900c8cb782ac49b4b701e5ca24fc7c2a (127.0.98.2:37125), 616163951943488284966a9bad264edf (127.0.98.3:34683)
I20260620 04:39:43.212355 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458954881ec84dd1b593820f142d6c70" candidate_uuid: "944bf06d10f64d98bc40b754a12fe9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a"
I20260620 04:39:43.212369 19988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458954881ec84dd1b593820f142d6c70" candidate_uuid: "944bf06d10f64d98bc40b754a12fe9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616163951943488284966a9bad264edf"
I20260620 04:39:43.212476 19988 raft_consensus.cc:3037] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:43.212476 19854 raft_consensus.cc:3037] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:43.213721 19988 raft_consensus.cc:2454] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944bf06d10f64d98bc40b754a12fe9ff in term 1.
I20260620 04:39:43.213721 19854 raft_consensus.cc:2454] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944bf06d10f64d98bc40b754a12fe9ff in term 1.
I20260620 04:39:43.214025 19653 leader_election.cc:302] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616163951943488284966a9bad264edf, 944bf06d10f64d98bc40b754a12fe9ff; no voters:
I20260620 04:39:43.214210 20057 raft_consensus.cc:2781] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:43.214418 20057 raft_consensus.cc:686] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 1 LEADER]: Becoming Leader. State: Replica: 944bf06d10f64d98bc40b754a12fe9ff, State: Running, Role: LEADER
I20260620 04:39:43.214718 20057 consensus_queue.cc:237] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } }
I20260620 04:39:43.217901 19573 catalog_manager.cc:5495] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 944bf06d10f64d98bc40b754a12fe9ff (127.0.98.1). New cstate: current_term: 1 leader_uuid: "944bf06d10f64d98bc40b754a12fe9ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944bf06d10f64d98bc40b754a12fe9ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20260620 04:39:43.243359 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.239423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.98.1:43203)
0620 04:39:43.239576 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.239584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.239602 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:39:43.239692 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.239695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.239748 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.239867 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:43.240140 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.240768 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.241807 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.242053 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.242168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.242489 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:43.242903 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:43.243056 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.243123 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.243181 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:39:43.249521 19854 raft_consensus.cc:1261] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 1 FOLLOWER]: Refusing update from remote peer 944bf06d10f64d98bc40b754a12fe9ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:43.249521 19988 raft_consensus.cc:1261] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 1 FOLLOWER]: Refusing update from remote peer 944bf06d10f64d98bc40b754a12fe9ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:43.250437 20062 consensus_queue.cc:1035] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c8cb782ac49b4b701e5ca24fc7c2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:43.250687 20057 consensus_queue.cc:1035] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "616163951943488284966a9bad264edf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:43.258613 20067 mvcc.cc:205] Tried to move back new op lower bound from 7298786849781616640 to 7298786849649827840. Current Snapshot: MvccSnapshot[applied={T|T < 7298786849781616640}]
I20260620 04:39:43.260545 20066 mvcc.cc:205] Tried to move back new op lower bound from 7298786849781616640 to 7298786849649827840. Current Snapshot: MvccSnapshot[applied={T|T < 7298786849781616640}]
I20260620 04:39:43.266422 19573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:43.266598 19573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:43.269042 19573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3c50fcb7cd346a692cca5fa8c303458: Sending DeleteTablet for 3 replicas of tablet 458954881ec84dd1b593820f142d6c70
I20260620 04:39:43.270088 19700 tablet_service.cc:1514] Processing DeleteTablet for tablet 458954881ec84dd1b593820f142d6c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34918
I20260620 04:39:43.270092 19834 tablet_service.cc:1514] Processing DeleteTablet for tablet 458954881ec84dd1b593820f142d6c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55034
I20260620 04:39:43.270233 19968 tablet_service.cc:1514] Processing DeleteTablet for tablet 458954881ec84dd1b593820f142d6c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53918
I20260620 04:39:43.270670 20075 tablet_replica.cc:330] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff: stopping tablet replica
I20260620 04:39:43.270709 20076 tablet_replica.cc:330] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: stopping tablet replica
I20260620 04:39:43.270989 20075 raft_consensus.cc:2229] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:39:43.270998 20076 raft_consensus.cc:2229] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:43.271147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19637
I20260620 04:39:43.271334 20075 raft_consensus.cc:2258] T 458954881ec84dd1b593820f142d6c70 P 944bf06d10f64d98bc40b754a12fe9ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:43.271699 20074 tablet_replica.cc:330] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: stopping tablet replica
I20260620 04:39:43.271991 20074 raft_consensus.cc:2229] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:43.272301 20074 raft_consensus.cc:2258] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:43.272406 20076 raft_consensus.cc:2258] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:43.274098 20076 ts_tablet_manager.cc:1905] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:43.274103 20074 ts_tablet_manager.cc:1905] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:43.276139 20076 ts_tablet_manager.cc:1918] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:43.276293 20076 log.cc:1196] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/458954881ec84dd1b593820f142d6c70
I20260620 04:39:43.276614 20076 ts_tablet_manager.cc:1939] T 458954881ec84dd1b593820f142d6c70 P 616163951943488284966a9bad264edf: Deleting consensus metadata
I20260620 04:39:43.277729 19559 catalog_manager.cc:4841] TS 616163951943488284966a9bad264edf (127.0.98.3:34683): tablet 458954881ec84dd1b593820f142d6c70 (table test-table [id=e94e63f1793c4edfaff03b72ff1a83a7]) successfully deleted
I20260620 04:39:43.278165 20074 ts_tablet_manager.cc:1918] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:43.278283 20074 log.cc:1196] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/458954881ec84dd1b593820f142d6c70
I20260620 04:39:43.278661 20074 ts_tablet_manager.cc:1939] T 458954881ec84dd1b593820f142d6c70 P 900c8cb782ac49b4b701e5ca24fc7c2a: Deleting consensus metadata
I20260620 04:39:43.280073 19562 catalog_manager.cc:4841] TS 900c8cb782ac49b4b701e5ca24fc7c2a (127.0.98.2:37125): tablet 458954881ec84dd1b593820f142d6c70 (table test-table [id=e94e63f1793c4edfaff03b72ff1a83a7]) successfully deleted
W20260620 04:39:43.280398 19559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43203 (error 108)
I20260620 04:39:43.280681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19771
I20260620 04:39:43.282038 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.281676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43203 (local address 127.0.0.1:34938)
0620 04:39:43.281864 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.281935 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 04:39:43.282220 19559 catalog_manager.cc:4584] TS 944bf06d10f64d98bc40b754a12fe9ff (127.0.98.1:43203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 458954881ec84dd1b593820f142d6c70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43203: connect: Connection refused (error 111)
I20260620 04:39:43.286885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19905
I20260620 04:39:43.292308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 19543
2026-06-20T04:39:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2283 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T04:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:43Z Disabled control of system clock
I20260620 04:39:43.324468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44859
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:43.438714 20081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:43.439035 20081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:43.439087 20081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:43.443269 20081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:43.443332 20081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:43.443349 20081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:43.443367 20081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:43.443382 20081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:43.448168 20081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:43.449280 20081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:43.450565 20081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:43.459240 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.459259 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:43.459466 20081 server_base.cc:1034] running on GCE node
W20260620 04:39:43.459714 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:43.460294 20081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:43.461326 20081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:43.462548 20081 hybrid_clock.cc:648] HybridClock initialized: now 1781930383462519 us; error 33 us; skew 500 ppm
I20260620 04:39:43.464428 20081 webserver.cc:460] Webserver started at http://127.0.98.62:39315/ using document root <none> and password file <none>
I20260620 04:39:43.464977 20081 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:43.465065 20081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:43.465281 20081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:43.467397 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "fc40607948fa4ab5bfbc27da574f1a42"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "25bd4995e48482ff280fc9d18fd146b0"
server_key_iv: "327d8acfd7c70ae1a9fe142c45ff302a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.468094 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc40607948fa4ab5bfbc27da574f1a42"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "25bd4995e48482ff280fc9d18fd146b0"
server_key_iv: "327d8acfd7c70ae1a9fe142c45ff302a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.473102 20081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:43.476702 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.479074 20081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:39:43.479257 20081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "fc40607948fa4ab5bfbc27da574f1a42"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "25bd4995e48482ff280fc9d18fd146b0"
server_key_iv: "327d8acfd7c70ae1a9fe142c45ff302a"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.479458 20081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:43.509847 20081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:43.510588 20081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:43.510785 20081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:43.518878 20081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44859
I20260620 04:39:43.518887 20147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44859 every 8 connection(s)
I20260620 04:39:43.519901 20081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:43.520517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20081
I20260620 04:39:43.520682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:43.521036 392 external_mini_cluster.cc:1428] Setting key 0f9763bfceaea8d50225e3fba5fb6c9a
I20260620 04:39:43.523795 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:43.529225 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: Bootstrap starting.
I20260620 04:39:43.530366 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.522740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.98.62:44859)
0620 04:39:43.523352 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.523369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.523402 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:39:43.524119 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.524135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.524578 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.524899 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:43.525322 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.526292 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.527167 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.527683 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.527853 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.528781 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:43.528786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:43.528797 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:43.529161 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:43.529213 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:43.529233 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:43.529360 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.529474 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.529718 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260620 04:39:43.531478 20148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:43.532248 20148 log.cc:824] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:43.534072 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: No bootstrap required, opened a new log
I20260620 04:39:43.537716 20148 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } }
I20260620 04:39:43.537968 20148 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:43.538005 20148 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc40607948fa4ab5bfbc27da574f1a42, State: Initialized, Role: FOLLOWER
I20260620 04:39:43.538340 20148 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } }
I20260620 04:39:43.538445 20148 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:43.538482 20148 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:43.538542 20148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:43.539489 20148 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } }
I20260620 04:39:43.539860 20148 leader_election.cc:302] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc40607948fa4ab5bfbc27da574f1a42; no voters:
I20260620 04:39:43.540118 20148 leader_election.cc:288] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:43.540284 20153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:43.540647 20153 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [term 1 LEADER]: Becoming Leader. State: Replica: fc40607948fa4ab5bfbc27da574f1a42, State: Running, Role: LEADER
I20260620 04:39:43.540962 20153 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } }
I20260620 04:39:43.541383 20148 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:43.542685 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc40607948fa4ab5bfbc27da574f1a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } } }
I20260620 04:39:43.542806 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:43.542757 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc40607948fa4ab5bfbc27da574f1a42. Latest consensus state: current_term: 1 leader_uuid: "fc40607948fa4ab5bfbc27da574f1a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc40607948fa4ab5bfbc27da574f1a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44859 } } }
I20260620 04:39:43.542891 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:43.543349 20162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:43.546519 20162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:43.552472 20162 catalog_manager.cc:1347] Generated new cluster ID: 18bcb37733ce4b13abee48722885ceab
I20260620 04:39:43.552569 20162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:43.562445 20162 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:43.563333 20162 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:43.573268 20162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: Generated new TSK 0
I20260620 04:39:43.573927 20162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:43.585263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--builtin_ntp_servers=127.0.98.20:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:39:43.698385 20172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:43.698652 20172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:43.698693 20172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:43.703022 20172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:43.703083 20172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:43.703140 20172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:39:43.708138 20172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:43.709151 20172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:43.710434 20172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:43.720327 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.720330 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.720330 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:43.721347 20172 server_base.cc:1034] running on GCE node
I20260620 04:39:43.721784 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:43.722431 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:43.723630 20172 hybrid_clock.cc:648] HybridClock initialized: now 1781930383723575 us; error 72 us; skew 500 ppm
I20260620 04:39:43.725509 20172 webserver.cc:460] Webserver started at http://127.0.98.1:44069/ using document root <none> and password file <none>
I20260620 04:39:43.726086 20172 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:43.726186 20172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:43.726424 20172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:43.728577 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data/instance:
uuid: "63d3cdf54b5749eca01aff011e7996bc"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "92a555765da8e1186631bed3dcf57cad"
server_key_iv: "a0a7b399c68e96c9e0a054477da6ce79"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.729141 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63d3cdf54b5749eca01aff011e7996bc"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "92a555765da8e1186631bed3dcf57cad"
server_key_iv: "a0a7b399c68e96c9e0a054477da6ce79"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.732918 20172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:39:43.735368 20186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.736622 20172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 04:39:43.736764 20172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal
uuid: "63d3cdf54b5749eca01aff011e7996bc"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "92a555765da8e1186631bed3dcf57cad"
server_key_iv: "a0a7b399c68e96c9e0a054477da6ce79"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.736876 20172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:43.778345 20172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:43.779111 20172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:43.779330 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:43.780341 20172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:43.781880 20172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:43.781949 20172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.782022 20172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:43.782053 20172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.791754 20172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43987
I20260620 04:39:43.791785 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43987 every 8 connection(s)
I20260620 04:39:43.792721 20172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:39:43.801918 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.794705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50899 (local address 127.0.98.62:44859)
0620 04:39:43.794885 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.794890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.795865 (+ 975us) server_negotiation.cc:407] Connection header received
0620 04:39:43.797304 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.797308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.797369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.797478 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:43.798312 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.798847 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.799961 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.800179 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.800348 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.801420 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:43.801423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:43.801425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:43.801436 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:43.801454 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:43.801462 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:43.801544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:43.801697 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:39:43.801762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:39:43.802284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20172
I20260620 04:39:43.802414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:39:43.802698 392 external_mini_cluster.cc:1428] Setting key b88f7f5c7782cb324c1b94f9f6df5687
I20260620 04:39:43.803582 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.795073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44859 (local address 127.0.98.1:50899)
0620 04:39:43.795710 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.795757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.797094 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.797635 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.797641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.797718 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:43.798171 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.798183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.798975 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.798983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.799828 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.799837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.800384 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.800387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:43.800546 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:43.801122 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:43.801124 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:43.801162 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:43.801230 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:43.801547 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:43.801558 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:43.801564 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:39:43.801697 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 04:39:43.801966 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":102,"threads_started":1}
I20260620 04:39:43.804721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--builtin_ntp_servers=127.0.98.20:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:43.804862 20300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44859
I20260620 04:39:43.805074 20300 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:43.805569 20300 heartbeater.cc:511] Master 127.0.98.62:44859 requested a full tablet report, sending...
I20260620 04:39:43.807142 20112 ts_manager.cc:194] Registered new tserver with Master: 63d3cdf54b5749eca01aff011e7996bc (127.0.98.1:43987)
I20260620 04:39:43.808786 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50899
W20260620 04:39:43.920833 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:43.921094 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:43.921133 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:43.925180 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:43.925238 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:43.925294 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:39:43.930002 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:43.931011 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:43.932382 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:43.941999 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.942131 20311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:43.942014 20309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:43.942267 20303 server_base.cc:1034] running on GCE node
I20260620 04:39:43.942739 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:43.943434 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:43.944643 20303 hybrid_clock.cc:648] HybridClock initialized: now 1781930383944611 us; error 43 us; skew 500 ppm
I20260620 04:39:43.946609 20303 webserver.cc:460] Webserver started at http://127.0.98.2:38363/ using document root <none> and password file <none>
I20260620 04:39:43.947185 20303 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:43.947278 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:43.947511 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:43.949559 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data/instance:
uuid: "e273317158d4486d8c7a87e3903d50c9"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "5f70c67a0805eaa08e14bca1996b89b0"
server_key_iv: "4fc1a12b3c9de9c2380cd927d8b7b171"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.950119 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e273317158d4486d8c7a87e3903d50c9"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "5f70c67a0805eaa08e14bca1996b89b0"
server_key_iv: "4fc1a12b3c9de9c2380cd927d8b7b171"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.954294 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:39:43.957330 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.958590 20303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:39:43.958748 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal
uuid: "e273317158d4486d8c7a87e3903d50c9"
format_stamp: "Formatted at 2026-06-20 04:39:43 on dist-test-slave-zmf4"
server_key: "5f70c67a0805eaa08e14bca1996b89b0"
server_key_iv: "4fc1a12b3c9de9c2380cd927d8b7b171"
server_key_version: "encryptionkey@0"
I20260620 04:39:43.958874 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:43.979470 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:43.980144 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:43.980357 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:43.980933 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:43.981905 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:43.981985 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.982059 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:43.982102 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:43.992025 20303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42575
I20260620 04:39:43.992054 20430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42575 every 8 connection(s)
I20260620 04:39:43.992985 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:39:44.001971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20303
I20260620 04:39:44.002149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:39:44.002465 392 external_mini_cluster.cc:1428] Setting key 755aec50222fc08aa43e968bb341a39a
I20260620 04:39:44.002544 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.994912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44491 (local address 127.0.98.62:44859)
0620 04:39:43.995060 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:43.995065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:43.996090 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 04:39:43.997519 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:43.997522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:43.997568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:43.997651 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:43.998483 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.999063 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.000253 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.000506 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.000608 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.001801 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:44.001807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:44.001810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:44.001825 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:44.001850 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:44.001861 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:44.001958 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.002239 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.002379 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:39:44.003378 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:43.995305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44859 (local address 127.0.98.2:44491)
0620 04:39:43.995953 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:43.995994 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:43.997329 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:43.997793 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:43.997799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:43.997875 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:43.998336 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:43.998347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:43.999184 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:43.999187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.000047 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.000056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.000676 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.000679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.000851 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.001483 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:44.001484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:44.001523 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:44.001585 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:44.002006 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:44.002015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:44.002020 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.002234 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.002557 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260620 04:39:44.004415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--builtin_ntp_servers=127.0.98.20:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:39:44.004639 20431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44859
I20260620 04:39:44.004889 20431 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:44.005414 20431 heartbeater.cc:511] Master 127.0.98.62:44859 requested a full tablet report, sending...
I20260620 04:39:44.006453 20112 ts_manager.cc:194] Registered new tserver with Master: e273317158d4486d8c7a87e3903d50c9 (127.0.98.2:42575)
I20260620 04:39:44.007050 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44491
W20260620 04:39:44.114711 20434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:44.114959 20434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:44.114996 20434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:44.119455 20434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:44.119515 20434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:44.119571 20434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:39:44.124415 20434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:44.125433 20434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:44.126756 20434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:44.136121 20439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:44.136123 20442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:44.136086 20440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:44.136286 20434 server_base.cc:1034] running on GCE node
I20260620 04:39:44.136734 20434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:44.137364 20434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:44.138530 20434 hybrid_clock.cc:648] HybridClock initialized: now 1781930384138516 us; error 28 us; skew 500 ppm
I20260620 04:39:44.140484 20434 webserver.cc:460] Webserver started at http://127.0.98.3:46775/ using document root <none> and password file <none>
I20260620 04:39:44.141026 20434 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:44.141086 20434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:44.141283 20434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:44.143426 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data/instance:
uuid: "5685d323e45443538de475a51b4a6466"
format_stamp: "Formatted at 2026-06-20 04:39:44 on dist-test-slave-zmf4"
server_key: "b189e7a33137b7d496ac8eca62146966"
server_key_iv: "cc16ad46c9f10276de841e2165358590"
server_key_version: "encryptionkey@0"
I20260620 04:39:44.143982 20434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5685d323e45443538de475a51b4a6466"
format_stamp: "Formatted at 2026-06-20 04:39:44 on dist-test-slave-zmf4"
server_key: "b189e7a33137b7d496ac8eca62146966"
server_key_iv: "cc16ad46c9f10276de841e2165358590"
server_key_version: "encryptionkey@0"
I20260620 04:39:44.147626 20434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:39:44.150099 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:44.151340 20434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:44.151484 20434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal
uuid: "5685d323e45443538de475a51b4a6466"
format_stamp: "Formatted at 2026-06-20 04:39:44 on dist-test-slave-zmf4"
server_key: "b189e7a33137b7d496ac8eca62146966"
server_key_iv: "cc16ad46c9f10276de841e2165358590"
server_key_version: "encryptionkey@0"
I20260620 04:39:44.151595 20434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:44.186950 20434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:44.187703 20434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:44.187925 20434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:44.188591 20434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:39:44.189602 20434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:39:44.189679 20434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:44.189757 20434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:39:44.189812 20434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:44.198908 20434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36631
I20260620 04:39:44.198932 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36631 every 8 connection(s)
I20260620 04:39:44.199764 20434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:39:44.201344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20434
I20260620 04:39:44.201460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:39:44.201767 392 external_mini_cluster.cc:1428] Setting key 9ba3cd891b1d9dfebc86a4e0483e434c
I20260620 04:39:44.208979 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.201544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49349 (local address 127.0.98.62:44859)
0620 04:39:44.201687 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.201692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.202860 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 04:39:44.204269 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.204272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.204327 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.204428 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:44.205284 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.205803 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.206919 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.207142 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.207255 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.208468 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:44.208472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:44.208474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:44.208494 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:44.208534 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:44.208543 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:44.208607 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.208770 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.208829 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:39:44.209769 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.202024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44859 (local address 127.0.98.3:49349)
0620 04:39:44.202713 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.202770 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.204043 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.204600 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.204606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.204685 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:39:44.205146 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.205157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.205930 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.205933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.206788 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.206797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.207290 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.207294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.207448 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.208161 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:39:44.208162 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:39:44.208244 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:39:44.208311 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:39:44.208620 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:39:44.208629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:39:44.208635 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.208759 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.209007 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260620 04:39:44.210817 20562 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44859
I20260620 04:39:44.211077 20562 heartbeater.cc:464] Registering TS with master...
I20260620 04:39:44.211596 20562 heartbeater.cc:511] Master 127.0.98.62:44859 requested a full tablet report, sending...
I20260620 04:39:44.212706 20112 ts_manager.cc:194] Registered new tserver with Master: 5685d323e45443538de475a51b4a6466 (127.0.98.3:36631)
I20260620 04:39:44.213317 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49349
I20260620 04:39:44.215399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:39:44.221369 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.217583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.98.62:44859)
0620 04:39:44.217727 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.217733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.217856 (+ 123us) server_negotiation.cc:407] Connection header received
0620 04:39:44.217947 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.217949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.217985 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.218051 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:44.218273 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.218737 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.219657 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.219869 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.219955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.220798 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:44.220801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:44.220803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:44.220814 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:44.220832 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:44.220842 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:44.220909 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.221044 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.221120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:39:44.224387 20112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:39:44.227210 20112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:39:44.242311 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.236767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42575 (local address 127.0.0.1:46668)
0620 04:39:44.237191 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.237205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.237402 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.237939 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.237943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.237970 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:44.238112 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.238117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.239510 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.239513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.240741 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.240750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.241703 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.241706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.241805 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.241891 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.242039 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.242152 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":99,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260620 04:39:44.242312 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.236458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.0.98.2:42575)
0620 04:39:44.236767 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.236779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.237419 (+ 640us) server_negotiation.cc:407] Connection header received
0620 04:39:44.237507 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.237511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.237651 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.237804 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:44.238222 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.239309 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.240886 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.241578 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.241692 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.241952 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.242067 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.242154 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260620 04:39:44.242682 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.235620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36631 (local address 127.0.0.1:40212)
0620 04:39:44.236781 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.236796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.236924 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.237673 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.237678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.237698 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:44.237821 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.237828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.239196 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.239199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.240760 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.240770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.242197 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.242200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.242384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.242409 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.242493 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.242560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":143,"thread_start_us":67,"threads_started":1}
I20260620 04:39:44.242946 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.235987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.0.98.3:36631)
0620 04:39:44.236435 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.236453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.236893 (+ 440us) server_negotiation.cc:407] Connection header received
0620 04:39:44.237191 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.237201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.237494 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.237660 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:44.237957 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.239089 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.240907 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.241808 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.241895 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.242097 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.242647 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.242754 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":134,"threads_started":1}
I20260620 04:39:44.243152 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.235593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43987 (local address 127.0.0.1:33278)
0620 04:39:44.236424 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.236477 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.237187 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.237939 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.237943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.237981 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:44.238098 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.238105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.239037 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.239043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.240760 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.240770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.242204 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.242207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.242834 (+ 627us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.242856 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.242941 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.243011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":667,"spinlock_wait_cycles":38784,"thread_start_us":107,"threads_started":1}
I20260620 04:39:44.243609 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.235958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.0.98.1:43987)
0620 04:39:44.236597 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.236605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.236631 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:39:44.237463 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.237469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.237642 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.237804 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:44.238252 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.238918 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.240942 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.241873 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.242007 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.242156 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.243127 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.243454 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":87,"threads_started":1}
I20260620 04:39:44.245769 20234 tablet_service.cc:1467] Processing CreateTablet for tablet 773fa8f1d3a14af4b26849836264d84f (DEFAULT_TABLE table=test-table [id=2381246b45be46e5a303412b41c4e693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:44.245589 20365 tablet_service.cc:1467] Processing CreateTablet for tablet 773fa8f1d3a14af4b26849836264d84f (DEFAULT_TABLE table=test-table [id=2381246b45be46e5a303412b41c4e693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:44.245926 20496 tablet_service.cc:1467] Processing CreateTablet for tablet 773fa8f1d3a14af4b26849836264d84f (DEFAULT_TABLE table=test-table [id=2381246b45be46e5a303412b41c4e693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:39:44.246661 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773fa8f1d3a14af4b26849836264d84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:44.246683 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773fa8f1d3a14af4b26849836264d84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:44.246692 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773fa8f1d3a14af4b26849836264d84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:44.252636 20579 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Bootstrap starting.
I20260620 04:39:44.254217 20580 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Bootstrap starting.
I20260620 04:39:44.255089 20579 tablet_bootstrap.cc:654] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:44.255810 20579 log.cc:824] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:44.255962 20580 tablet_bootstrap.cc:654] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:44.256567 20581 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Bootstrap starting.
I20260620 04:39:44.256826 20580 log.cc:824] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:44.257937 20579 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: No bootstrap required, opened a new log
I20260620 04:39:44.258082 20579 ts_tablet_manager.cc:1397] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:39:44.258464 20581 tablet_bootstrap.cc:654] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:44.259073 20581 log.cc:824] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:44.260890 20581 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: No bootstrap required, opened a new log
I20260620 04:39:44.261096 20581 ts_tablet_manager.cc:1397] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:39:44.261418 20580 tablet_bootstrap.cc:492] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: No bootstrap required, opened a new log
I20260620 04:39:44.261608 20580 ts_tablet_manager.cc:1397] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260620 04:39:44.261452 20579 raft_consensus.cc:348] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.261711 20579 raft_consensus.cc:374] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:44.261765 20579 raft_consensus.cc:729] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d3cdf54b5749eca01aff011e7996bc, State: Initialized, Role: FOLLOWER
I20260620 04:39:44.262269 20579 consensus_queue.cc:260] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.262883 20300 heartbeater.cc:503] Master 127.0.98.62:44859 was elected leader, sending a full tablet report...
I20260620 04:39:44.263233 20579 ts_tablet_manager.cc:1428] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 04:39:44.265266 20580 raft_consensus.cc:348] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.265516 20580 raft_consensus.cc:374] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:44.265568 20580 raft_consensus.cc:729] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e273317158d4486d8c7a87e3903d50c9, State: Initialized, Role: FOLLOWER
I20260620 04:39:44.265470 20581 raft_consensus.cc:348] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.265678 20581 raft_consensus.cc:374] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:44.265718 20581 raft_consensus.cc:729] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5685d323e45443538de475a51b4a6466, State: Initialized, Role: FOLLOWER
I20260620 04:39:44.265971 20580 consensus_queue.cc:260] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.266050 20581 consensus_queue.cc:260] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.266562 20431 heartbeater.cc:503] Master 127.0.98.62:44859 was elected leader, sending a full tablet report...
I20260620 04:39:44.266865 20580 ts_tablet_manager.cc:1428] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:39:44.266973 20562 heartbeater.cc:503] Master 127.0.98.62:44859 was elected leader, sending a full tablet report...
I20260620 04:39:44.267886 20581 ts_tablet_manager.cc:1428] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 04:39:44.294003 20301 tablet.cc:2369] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:39:44.356129 20585 raft_consensus.cc:482] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:39:44.356420 20585 raft_consensus.cc:504] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.357640 20585 leader_election.cc:288] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5685d323e45443538de475a51b4a6466 (127.0.98.3:36631), e273317158d4486d8c7a87e3903d50c9 (127.0.98.2:42575)
I20260620 04:39:44.362478 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.358409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35287 (local address 127.0.98.2:42575)
0620 04:39:44.358529 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.358534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.358663 (+ 129us) server_negotiation.cc:407] Connection header received
0620 04:39:44.358759 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.358761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.358830 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.358910 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:44.359146 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.359833 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.361400 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.361895 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.361994 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.362052 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.362255 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.362325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:39:44.362483 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.358270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42575 (local address 127.0.98.1:35287)
0620 04:39:44.358569 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.358582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.358672 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.358932 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.358934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.358949 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:44.359044 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.359048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.359934 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.359937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.361245 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.361263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.362006 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.362010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.362124 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.362147 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.362224 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.362303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":59,"thread_start_us":106,"threads_started":1}
I20260620 04:39:44.362624 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.358157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34095 (local address 127.0.98.3:36631)
0620 04:39:44.358294 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.358299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.358316 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:39:44.358378 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.358382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.358430 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.358517 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:39:44.358835 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.359642 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.361433 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.362041 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.362141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.362197 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.362430 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.362503 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:39:44.363011 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.357891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36631 (local address 127.0.98.1:34095)
0620 04:39:44.358139 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.358156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:39:44.358265 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:39:44.358540 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:39:44.358545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:39:44.358562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:39:44.358711 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.358721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.359773 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.359776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.361245 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:39:44.361257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.362211 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.362217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:39:44.362308 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.362327 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:39:44.362396 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:39:44.362445 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":72,"thread_start_us":95,"threads_started":1}
I20260620 04:39:44.363250 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773fa8f1d3a14af4b26849836264d84f" candidate_uuid: "63d3cdf54b5749eca01aff011e7996bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e273317158d4486d8c7a87e3903d50c9" is_pre_election: true
I20260620 04:39:44.363598 20385 raft_consensus.cc:2454] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d3cdf54b5749eca01aff011e7996bc in term 0.
I20260620 04:39:44.363601 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773fa8f1d3a14af4b26849836264d84f" candidate_uuid: "63d3cdf54b5749eca01aff011e7996bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5685d323e45443538de475a51b4a6466" is_pre_election: true
I20260620 04:39:44.363852 20516 raft_consensus.cc:2454] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d3cdf54b5749eca01aff011e7996bc in term 0.
I20260620 04:39:44.364257 20190 leader_election.cc:302] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d3cdf54b5749eca01aff011e7996bc, e273317158d4486d8c7a87e3903d50c9; no voters:
I20260620 04:39:44.364732 20585 raft_consensus.cc:2781] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:39:44.364838 20585 raft_consensus.cc:482] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:39:44.364900 20585 raft_consensus.cc:3037] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:44.365916 20585 raft_consensus.cc:504] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.366240 20585 leader_election.cc:288] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [CANDIDATE]: Term 1 election: Requested vote from peers 5685d323e45443538de475a51b4a6466 (127.0.98.3:36631), e273317158d4486d8c7a87e3903d50c9 (127.0.98.2:42575)
I20260620 04:39:44.366554 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773fa8f1d3a14af4b26849836264d84f" candidate_uuid: "63d3cdf54b5749eca01aff011e7996bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5685d323e45443538de475a51b4a6466"
I20260620 04:39:44.366622 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773fa8f1d3a14af4b26849836264d84f" candidate_uuid: "63d3cdf54b5749eca01aff011e7996bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e273317158d4486d8c7a87e3903d50c9"
I20260620 04:39:44.366688 20516 raft_consensus.cc:3037] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:44.366762 20385 raft_consensus.cc:3037] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:44.367720 20516 raft_consensus.cc:2454] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d3cdf54b5749eca01aff011e7996bc in term 1.
I20260620 04:39:44.367738 20385 raft_consensus.cc:2454] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d3cdf54b5749eca01aff011e7996bc in term 1.
I20260620 04:39:44.368023 20190 leader_election.cc:302] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5685d323e45443538de475a51b4a6466, 63d3cdf54b5749eca01aff011e7996bc; no voters:
I20260620 04:39:44.368224 20585 raft_consensus.cc:2781] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:44.368431 20585 raft_consensus.cc:686] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [term 1 LEADER]: Becoming Leader. State: Replica: 63d3cdf54b5749eca01aff011e7996bc, State: Running, Role: LEADER
I20260620 04:39:44.368739 20585 consensus_queue.cc:237] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } }
I20260620 04:39:44.371953 20111 catalog_manager.cc:5495] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d3cdf54b5749eca01aff011e7996bc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "63d3cdf54b5749eca01aff011e7996bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d3cdf54b5749eca01aff011e7996bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 } health_report { overall_health: UNKNOWN } } }
I20260620 04:39:44.400403 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.396483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.98.1:43987)
0620 04:39:44.396651 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:44.396656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:44.396670 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:39:44.396809 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:44.396816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:44.396872 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:44.396986 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:39:44.397230 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.397852 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.398833 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:44.399072 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:44.399198 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:44.399527 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:39:44.399955 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:39:44.400100 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:44.400171 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:39:44.400247 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:39:44.406152 20516 raft_consensus.cc:1261] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 1 FOLLOWER]: Refusing update from remote peer 63d3cdf54b5749eca01aff011e7996bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:44.406176 20385 raft_consensus.cc:1261] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 1 FOLLOWER]: Refusing update from remote peer 63d3cdf54b5749eca01aff011e7996bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:39:44.406823 20585 consensus_queue.cc:1035] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5685d323e45443538de475a51b4a6466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:44.407040 20590 consensus_queue.cc:1035] T 773fa8f1d3a14af4b26849836264d84f P 63d3cdf54b5749eca01aff011e7996bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e273317158d4486d8c7a87e3903d50c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:39:44.415329 20595 mvcc.cc:205] Tried to move back new op lower bound from 7298786854520107008 to 7298786854377345024. Current Snapshot: MvccSnapshot[applied={T|T < 7298786854520107008}]
I20260620 04:39:44.417001 20594 mvcc.cc:205] Tried to move back new op lower bound from 7298786854520107008 to 7298786854377345024. Current Snapshot: MvccSnapshot[applied={T|T < 7298786854520107008}]
I20260620 04:39:44.422181 20112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:44.422349 20112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:39:44.424953 20112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc40607948fa4ab5bfbc27da574f1a42: Sending DeleteTablet for 3 replicas of tablet 773fa8f1d3a14af4b26849836264d84f
I20260620 04:39:44.425853 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 773fa8f1d3a14af4b26849836264d84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:44 UTC) from {username='slave'} at 127.0.0.1:40212
I20260620 04:39:44.425829 20234 tablet_service.cc:1514] Processing DeleteTablet for tablet 773fa8f1d3a14af4b26849836264d84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:44 UTC) from {username='slave'} at 127.0.0.1:33278
I20260620 04:39:44.425997 20365 tablet_service.cc:1514] Processing DeleteTablet for tablet 773fa8f1d3a14af4b26849836264d84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:39:44 UTC) from {username='slave'} at 127.0.0.1:46668
I20260620 04:39:44.426482 20604 tablet_replica.cc:330] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: stopping tablet replica
I20260620 04:39:44.426513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20172
I20260620 04:39:44.426558 20602 tablet_replica.cc:330] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: stopping tablet replica
I20260620 04:39:44.427726 20604 raft_consensus.cc:2229] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:44.427726 20602 raft_consensus.cc:2229] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:39:44.428043 20604 raft_consensus.cc:2258] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:44.428045 20602 raft_consensus.cc:2258] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:39:44.429543 20602 ts_tablet_manager.cc:1905] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:44.429602 20604 ts_tablet_manager.cc:1905] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:39:44.431772 20602 ts_tablet_manager.cc:1918] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:44.431908 20602 log.cc:1196] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-2/wal/wals/773fa8f1d3a14af4b26849836264d84f
I20260620 04:39:44.432260 20602 ts_tablet_manager.cc:1939] T 773fa8f1d3a14af4b26849836264d84f P 5685d323e45443538de475a51b4a6466: Deleting consensus metadata
I20260620 04:39:44.432869 20604 ts_tablet_manager.cc:1918] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:39:44.432969 20604 log.cc:1196] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Deleting WAL directory at /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781930128116734-392-0/minicluster-data/ts-1/wal/wals/773fa8f1d3a14af4b26849836264d84f
I20260620 04:39:44.433394 20604 ts_tablet_manager.cc:1939] T 773fa8f1d3a14af4b26849836264d84f P e273317158d4486d8c7a87e3903d50c9: Deleting consensus metadata
I20260620 04:39:44.433480 20099 catalog_manager.cc:4841] TS 5685d323e45443538de475a51b4a6466 (127.0.98.3:36631): tablet 773fa8f1d3a14af4b26849836264d84f (table test-table [id=2381246b45be46e5a303412b41c4e693]) successfully deleted
I20260620 04:39:44.434520 20099 catalog_manager.cc:4841] TS e273317158d4486d8c7a87e3903d50c9 (127.0.98.2:42575): tablet 773fa8f1d3a14af4b26849836264d84f (table test-table [id=2381246b45be46e5a303412b41c4e693]) successfully deleted
W20260620 04:39:44.434599 20097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43987 (error 108)
I20260620 04:39:44.435304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20303
I20260620 04:39:44.435606 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:44.435300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43987 (local address 127.0.0.1:33288)
0620 04:39:44.435428 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 04:39:44.435515 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:39:44.436815 20097 catalog_manager.cc:4584] TS 63d3cdf54b5749eca01aff011e7996bc (127.0.98.1:43987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 773fa8f1d3a14af4b26849836264d84f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43987: connect: Connection refused (error 111)
I20260620 04:39:44.441457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20434
I20260620 04:39:44.447165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20081
2026-06-20T04:39:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1153 ms)
[----------] 6 tests from EncryptionPolicyTest (10547 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:44 dist-test-slave-zmf4 krb5kdc[20608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:44 dist-test-slave-zmf4 krb5kdc[20608](info): set up 2 sockets
Jun 20 04:39:44 dist-test-slave-zmf4 krb5kdc[20608](info): commencing operation
krb5kdc: starting...
W20260620 04:39:45.524302 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:45.649582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:45.761399 20621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:45.761706 20621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:45.761790 20621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:45.766167 20621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:45.766244 20621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:45.766288 20621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:45.766347 20621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:45.766387 20621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:45.771679 20621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:45.772764 20621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:45.773761 20621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:45.781791 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:45.781827 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:45.781801 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:45.781867 20621 server_base.cc:1034] running on GCE node
I20260620 04:39:45.782678 20621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:45.783782 20621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:45.784986 20621 hybrid_clock.cc:648] HybridClock initialized: now 1781930385784966 us; error 33 us; skew 500 ppm
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:45.792639 20621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:45.793520 20621 webserver.cc:460] Webserver started at http://127.0.0.1:39917/ using document root <none> and password file <none>
I20260620 04:39:45.794078 20621 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:45.794138 20621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:45.794348 20621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:45.796480 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "7466800e846a494ca6b3708e34a21e21"
format_stamp: "Formatted at 2026-06-20 04:39:45 on dist-test-slave-zmf4"
server_key: "6e2fdce0bf724bf3e83c7a796c989acb"
server_key_iv: "2876a3ce40a180349313c2ac57bf350f"
server_key_version: "encryptionkey@0"
I20260620 04:39:45.797053 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "7466800e846a494ca6b3708e34a21e21"
format_stamp: "Formatted at 2026-06-20 04:39:45 on dist-test-slave-zmf4"
server_key: "6e2fdce0bf724bf3e83c7a796c989acb"
server_key_iv: "2876a3ce40a180349313c2ac57bf350f"
server_key_version: "encryptionkey@0"
I20260620 04:39:45.800637 20621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:45.802975 20636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:45.804312 20621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:45.804448 20621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "7466800e846a494ca6b3708e34a21e21"
format_stamp: "Formatted at 2026-06-20 04:39:45 on dist-test-slave-zmf4"
server_key: "6e2fdce0bf724bf3e83c7a796c989acb"
server_key_iv: "2876a3ce40a180349313c2ac57bf350f"
server_key_version: "encryptionkey@0"
I20260620 04:39:45.804562 20621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:45.816052 20621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:45.819011 20621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:45.819216 20621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:45.826490 20621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34203
I20260620 04:39:45.826529 20688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34203 every 8 connection(s)
I20260620 04:39:45.827392 20621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:45.830103 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:45.835286 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21: Bootstrap starting.
I20260620 04:39:45.836174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20621
I20260620 04:39:45.836371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:45.836650 392 external_mini_cluster.cc:1428] Setting key 4405f6ca955861d9c216505346b2b0e1
I20260620 04:39:45.837994 20689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:45.838707 20689 log.cc:824] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:45.840504 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21: No bootstrap required, opened a new log
I20260620 04:39:45.844646 20689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 04:39:45.844851 20689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:45.844884 20689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7466800e846a494ca6b3708e34a21e21, State: Initialized, Role: FOLLOWER
I20260620 04:39:45.845347 20689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 04:39:45.845466 20689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:45.845551 20689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:45.845672 20689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:39:45.846596 20689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 04:39:45.846942 20689 leader_election.cc:302] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7466800e846a494ca6b3708e34a21e21; no voters:
I20260620 04:39:45.847312 20689 leader_election.cc:288] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:45.847450 20694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:45.847672 20694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [term 1 LEADER]: Becoming Leader. State: Replica: 7466800e846a494ca6b3708e34a21e21, State: Running, Role: LEADER
I20260620 04:39:45.847987 20694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 04:39:45.848657 20689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:45.849509 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7466800e846a494ca6b3708e34a21e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 04:39:45.849617 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:45.849958 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7466800e846a494ca6b3708e34a21e21. Latest consensus state: current_term: 1 leader_uuid: "7466800e846a494ca6b3708e34a21e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7466800e846a494ca6b3708e34a21e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 04:39:45.850106 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:45.850961 20703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:45.852589 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:45.837708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56847 (local address 127.0.0.1:34203)
0620 04:39:45.838214 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:45.838230 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:45.838266 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:39:45.839049 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:45.839088 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:45.839453 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:45.839814 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:45.840662 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:45.842117 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:45.843046 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:45.843819 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:45.843977 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:45.846706 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:45.846712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:45.846728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:45.846756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:45.848927 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:45.849587 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:45.849595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:45.849603 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:45.849786 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:45.850198 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:45.850203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:45.850208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:45.850711 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:45.851562 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:45.851694 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 04:39:45.851864 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":134,"threads_started":1}
I20260620 04:39:45.854246 20703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:45.855227 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:45.859938 20703 catalog_manager.cc:1347] Generated new cluster ID: b02449dbc2a9486992a216317b7f6abe
I20260620 04:39:45.859999 20703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:45.865677 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:45.857139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37381 (local address 127.0.0.1:34203)
0620 04:39:45.857244 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:45.857248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:45.857463 (+ 215us) server_negotiation.cc:407] Connection header received
0620 04:39:45.857579 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:45.857582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:45.857621 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:45.857712 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:45.858486 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:45.859610 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:45.860640 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:45.860898 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:45.861011 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:45.862349 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:45.862352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:45.862355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:45.862380 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:45.863862 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:45.864413 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:45.864416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:45.864418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:45.864498 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:45.864830 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:45.864834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:45.864836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:45.865005 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:45.865159 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:45.865424 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 04:39:45.865514 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:39:45.891077 20703 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:45.892122 20703 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:45.902531 20703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7466800e846a494ca6b3708e34a21e21: Generated new TSK 0
I20260620 04:39:45.903234 20703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:45.909289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20621
W20260620 04:39:45.915089 20606 connection.cc:535] client connection to 127.0.0.1:34203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34203): Connection reset by peer (error 104)
2026-06-20T04:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20720](info): set up 2 sockets
Jun 20 04:39:45 dist-test-slave-zmf4 krb5kdc[20720](info): commencing operation
krb5kdc: starting...
W20260620 04:39:46.956499 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:47 dist-test-slave-zmf4 krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:47.084646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:47.199218 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:47.199493 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:47.199536 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:47.203864 20733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:47.203940 20733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:47.203960 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:47.203977 20733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:47.203992 20733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:47.209354 20733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:47.210417 20733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:47.211344 20733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:47.219894 20738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:47.219866 20741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:47.219982 20739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:47.220156 20733 server_base.cc:1034] running on GCE node
I20260620 04:39:47.220790 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:47.221796 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:47.222988 20733 hybrid_clock.cc:648] HybridClock initialized: now 1781930387222971 us; error 30 us; skew 500 ppm
Jun 20 04:39:47 dist-test-slave-zmf4 krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:47.225812 20733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:47.226672 20733 webserver.cc:460] Webserver started at http://127.0.0.1:35835/ using document root <none> and password file <none>
I20260620 04:39:47.227232 20733 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:47.227329 20733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:47.227552 20733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:47.229599 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "6f616899937c4120bd52eee75d20b7d6"
format_stamp: "Formatted at 2026-06-20 04:39:47 on dist-test-slave-zmf4"
server_key: "7abd70ef9680ca2244efe64f4ecd24bb"
server_key_iv: "017c62e823a751e0b212b7ed13bda59d"
server_key_version: "encryptionkey@0"
I20260620 04:39:47.230177 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f616899937c4120bd52eee75d20b7d6"
format_stamp: "Formatted at 2026-06-20 04:39:47 on dist-test-slave-zmf4"
server_key: "7abd70ef9680ca2244efe64f4ecd24bb"
server_key_iv: "017c62e823a751e0b212b7ed13bda59d"
server_key_version: "encryptionkey@0"
I20260620 04:39:47.234113 20733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 04:39:47.236939 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:47.238255 20733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:39:47.238408 20733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "6f616899937c4120bd52eee75d20b7d6"
format_stamp: "Formatted at 2026-06-20 04:39:47 on dist-test-slave-zmf4"
server_key: "7abd70ef9680ca2244efe64f4ecd24bb"
server_key_iv: "017c62e823a751e0b212b7ed13bda59d"
server_key_version: "encryptionkey@0"
I20260620 04:39:47.238535 20733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:47.271137 20733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:47.274412 20733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:47.274648 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:47.281822 20733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44713
I20260620 04:39:47.281849 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44713 every 8 connection(s)
I20260620 04:39:47.282748 20733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:47.285353 20801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:47.290539 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6: Bootstrap starting.
I20260620 04:39:47.291198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20733
I20260620 04:39:47.291342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:47.291639 392 external_mini_cluster.cc:1428] Setting key 50975ac5bcaae0086ec5cc6564e70e91
I20260620 04:39:47.293188 20801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:47.293913 20801 log.cc:824] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:47.295737 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6: No bootstrap required, opened a new log
I20260620 04:39:47.299309 20801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260620 04:39:47.299547 20801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:47.299654 20801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f616899937c4120bd52eee75d20b7d6, State: Initialized, Role: FOLLOWER
I20260620 04:39:47.300144 20801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260620 04:39:47.300323 20801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:47.300412 20801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:47.300518 20801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:47.300648 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:47.292891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41441 (local address 127.0.0.1:44713)
0620 04:39:47.293333 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:47.293351 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:47.293391 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:39:47.294049 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:47.294070 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:47.294474 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:47.294840 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:47.295219 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:47.296297 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:47.297178 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:47.297652 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:47.297828 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:47.298802 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:47.298807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:47.298819 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:47.299201 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:47.299252 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:47.299274 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:47.299438 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:47.299597 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 04:39:47.299731 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260620 04:39:47.301613 20801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260620 04:39:47.301997 20801 leader_election.cc:302] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f616899937c4120bd52eee75d20b7d6; no voters:
I20260620 04:39:47.302316 20801 leader_election.cc:288] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:47.302420 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:47.302642 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f616899937c4120bd52eee75d20b7d6, State: Running, Role: LEADER
I20260620 04:39:47.303004 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260620 04:39:47.303560 20801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:47.303941 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:47.304731 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f616899937c4120bd52eee75d20b7d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260620 04:39:47.304867 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:47.305154 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f616899937c4120bd52eee75d20b7d6. Latest consensus state: current_term: 1 leader_uuid: "6f616899937c4120bd52eee75d20b7d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f616899937c4120bd52eee75d20b7d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260620 04:39:47.305245 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:47.305513 20815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:47.309098 20815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:47.310451 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:47.306225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54723 (local address 127.0.0.1:44713)
0620 04:39:47.306357 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:47.306363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:47.306900 (+ 537us) server_negotiation.cc:407] Connection header received
0620 04:39:47.307016 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:47.307019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:47.307066 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:47.307152 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:47.307418 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:47.307922 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:47.308823 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:47.309046 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:47.309157 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:47.309998 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:47.310002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:47.310006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:47.310023 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:47.310045 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:47.310056 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:47.310151 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:47.310278 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 04:39:47.310319 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:39:47.314671 20815 catalog_manager.cc:1347] Generated new cluster ID: 17e3ab36aa664872b0b2dc9c7cf8ef06
I20260620 04:39:47.314738 20815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:47.339514 20815 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:47.340533 20815 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:47.346558 20815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f616899937c4120bd52eee75d20b7d6: Generated new TSK 0
I20260620 04:39:47.347211 20815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:47.360230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20733
W20260620 04:39:47.365664 20718 connection.cc:535] client connection to 127.0.0.1:44713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44713): Connection reset by peer (error 104)
2026-06-20T04:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:47 dist-test-slave-zmf4 krb5kdc[20832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:47 dist-test-slave-zmf4 krb5kdc[20832](info): set up 2 sockets
Jun 20 04:39:47 dist-test-slave-zmf4 krb5kdc[20832](info): commencing operation
krb5kdc: starting...
W20260620 04:39:48.396453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:48 dist-test-slave-zmf4 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:48.525666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:48.634395 20845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:48.634661 20845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:48.634703 20845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:48.638765 20845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:48.638828 20845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:48.638849 20845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:48.638865 20845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:48.639014 20845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:48.644455 20845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:48.645515 20845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:48.646473 20845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:48.654801 20850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:48.654825 20851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:48.655046 20853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:48.654924 20845 server_base.cc:1034] running on GCE node
I20260620 04:39:48.655727 20845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:48.656849 20845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:48.658025 20845 hybrid_clock.cc:648] HybridClock initialized: now 1781930388657921 us; error 109 us; skew 500 ppm
Jun 20 04:39:48 dist-test-slave-zmf4 krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:48.660974 20845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:48.661867 20845 webserver.cc:460] Webserver started at http://127.0.0.1:42303/ using document root <none> and password file <none>
I20260620 04:39:48.662421 20845 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:48.662482 20845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:48.662680 20845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:48.664716 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "5bbc34d8525648d38252b38f335a9634"
format_stamp: "Formatted at 2026-06-20 04:39:48 on dist-test-slave-zmf4"
server_key: "7fe5fd3eb2e4a12f54d9c1da9d86cb2c"
server_key_iv: "660574d39d0b6d6171eecd435276de45"
server_key_version: "encryptionkey@0"
I20260620 04:39:48.665275 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bbc34d8525648d38252b38f335a9634"
format_stamp: "Formatted at 2026-06-20 04:39:48 on dist-test-slave-zmf4"
server_key: "7fe5fd3eb2e4a12f54d9c1da9d86cb2c"
server_key_iv: "660574d39d0b6d6171eecd435276de45"
server_key_version: "encryptionkey@0"
I20260620 04:39:48.668948 20845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:48.671320 20860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:48.672704 20845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:48.672845 20845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "5bbc34d8525648d38252b38f335a9634"
format_stamp: "Formatted at 2026-06-20 04:39:48 on dist-test-slave-zmf4"
server_key: "7fe5fd3eb2e4a12f54d9c1da9d86cb2c"
server_key_iv: "660574d39d0b6d6171eecd435276de45"
server_key_version: "encryptionkey@0"
I20260620 04:39:48.672961 20845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:48.677459 20845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:48.680588 20845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:48.680768 20845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:48.687875 20845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41923
I20260620 04:39:48.687901 20912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41923 every 8 connection(s)
I20260620 04:39:48.688799 20845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:48.691349 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:48.691478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20845
I20260620 04:39:48.691618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:48.691907 392 external_mini_cluster.cc:1428] Setting key 55cfd71498ce8b057ef3ebf0b7ace106
I20260620 04:39:48.697371 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:48.693400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55461 (local address 127.0.0.1:41923)
0620 04:39:48.693857 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:48.693875 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:48.693911 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:39:48.694679 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:48.694697 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:48.695081 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:48.695436 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:48.696184 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:48.696246 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:48.696265 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:48.696276 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:48.696328 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:48.696354 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:48.696498 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:48.696566 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 04:39:48.696713 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260620 04:39:48.698339 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: Bootstrap starting.
I20260620 04:39:48.700217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:48.701020 20913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:48.701905 20913 log.cc:824] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:48.703680 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: No bootstrap required, opened a new log
I20260620 04:39:48.705510 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:48.702542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36717 (local address 127.0.0.1:41923)
0620 04:39:48.702654 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:48.702658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:48.704033 (+ 1375us) server_negotiation.cc:407] Connection header received
0620 04:39:48.704176 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:48.704179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:48.704271 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:48.704351 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:48.705082 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:48.705085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:48.705088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:48.705099 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:48.705125 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:48.705137 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:48.705230 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:48.705349 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 04:39:48.705385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:39:48.707530 20913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260620 04:39:48.707731 20913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:48.707794 20913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbc34d8525648d38252b38f335a9634, State: Initialized, Role: FOLLOWER
I20260620 04:39:48.708267 20913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260620 04:39:48.708431 20913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:48.708500 20913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:48.708596 20913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:48.709452 20913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260620 04:39:48.709784 20913 leader_election.cc:302] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbc34d8525648d38252b38f335a9634; no voters:
I20260620 04:39:48.710076 20913 leader_election.cc:288] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:48.710227 20923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:48.710486 20923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbc34d8525648d38252b38f335a9634, State: Running, Role: LEADER
I20260620 04:39:48.710764 20923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } }
I20260620 04:39:48.711231 20913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:48.712497 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbc34d8525648d38252b38f335a9634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } } }
I20260620 04:39:48.712634 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:48.712496 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbc34d8525648d38252b38f335a9634. Latest consensus state: current_term: 1 leader_uuid: "5bbc34d8525648d38252b38f335a9634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc34d8525648d38252b38f335a9634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41923 } } }
I20260620 04:39:48.713683 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634 [sys.catalog]: This master's current role is: LEADER
W20260620 04:39:48.715668 20938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:39:48.715729 20938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:39:48.715809 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:48.716589 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:48.721927 20933 catalog_manager.cc:1347] Generated new cluster ID: 34b98dfae3794305aa4b85d706eabc22
I20260620 04:39:48.721995 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:48.758143 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:48.758466 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:48.766005 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bbc34d8525648d38252b38f335a9634: Generated new TSK 0
I20260620 04:39:48.766688 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:48.781857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20845
2026-06-20T04:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:48 dist-test-slave-zmf4 krb5kdc[20944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:48 dist-test-slave-zmf4 krb5kdc[20944](info): set up 2 sockets
Jun 20 04:39:48 dist-test-slave-zmf4 krb5kdc[20944](info): commencing operation
krb5kdc: starting...
W20260620 04:39:49.844281 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:49 dist-test-slave-zmf4 krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:49.967309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:50.077594 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:50.077899 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:50.077981 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:50.082051 20957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:50.082140 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:50.082178 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:50.082235 20957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:50.082275 20957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:50.087884 20957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:50.089175 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:50.090298 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:50.098891 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:50.098886 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:50.098922 20957 server_base.cc:1034] running on GCE node
W20260620 04:39:50.098886 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:50.099655 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:50.100816 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:50.101986 20957 hybrid_clock.cc:648] HybridClock initialized: now 1781930390101958 us; error 44 us; skew 500 ppm
Jun 20 04:39:50 dist-test-slave-zmf4 krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:50.104758 20957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:50.105629 20957 webserver.cc:460] Webserver started at http://127.0.0.1:38999/ using document root <none> and password file <none>
I20260620 04:39:50.106174 20957 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:50.106267 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:50.106500 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:50.108533 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "93a3250e1c504c30b03b55e5f56e29c2"
format_stamp: "Formatted at 2026-06-20 04:39:50 on dist-test-slave-zmf4"
server_key: "92c1b848f8aecaedb21cf9ea8c085006"
server_key_iv: "10da612fdb590136b9847f87f8cd9d6d"
server_key_version: "encryptionkey@0"
I20260620 04:39:50.109084 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "93a3250e1c504c30b03b55e5f56e29c2"
format_stamp: "Formatted at 2026-06-20 04:39:50 on dist-test-slave-zmf4"
server_key: "92c1b848f8aecaedb21cf9ea8c085006"
server_key_iv: "10da612fdb590136b9847f87f8cd9d6d"
server_key_version: "encryptionkey@0"
I20260620 04:39:50.112777 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:50.115211 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:50.116593 20957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:50.116761 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "93a3250e1c504c30b03b55e5f56e29c2"
format_stamp: "Formatted at 2026-06-20 04:39:50 on dist-test-slave-zmf4"
server_key: "92c1b848f8aecaedb21cf9ea8c085006"
server_key_iv: "10da612fdb590136b9847f87f8cd9d6d"
server_key_version: "encryptionkey@0"
I20260620 04:39:50.116892 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:50.139979 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:50.143388 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:50.143610 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:50.150662 20957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36647
I20260620 04:39:50.150679 21024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36647 every 8 connection(s)
I20260620 04:39:50.151544 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:50.153358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 20957
I20260620 04:39:50.153513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:50.153820 392 external_mini_cluster.cc:1428] Setting key b8eb9262d284e0c79836d3c0a6227a2c
I20260620 04:39:50.154558 21025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:50.161249 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2: Bootstrap starting.
Jun 20 04:39:50 dist-test-slave-zmf4 krb5kdc[20944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:39:50.163528 21025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:50.164390 21025 log.cc:824] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:50.166697 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2: No bootstrap required, opened a new log
I20260620 04:39:50.169363 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:50.155135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45299 (local address 127.0.0.1:36647)
0620 04:39:50.155531 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:50.155550 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:50.155590 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:39:50.156445 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:50.156467 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:50.156884 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:50.157230 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:50.157937 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:50.159408 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:50.160397 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:50.161042 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:50.161197 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:50.164053 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:50.164059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:50.164074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:50.164103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:50.166208 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:50.166631 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:50.166636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:50.166641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:50.166776 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:50.167125 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:50.167128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:50.167130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:50.167598 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:50.167780 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:50.168062 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 04:39:50.168373 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260620 04:39:50.171072 21025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } }
I20260620 04:39:50.171342 21025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:50.171402 21025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a3250e1c504c30b03b55e5f56e29c2, State: Initialized, Role: FOLLOWER
I20260620 04:39:50.171834 21025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } }
I20260620 04:39:50.171953 21025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:50.172034 21025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:50.172132 21025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:50.172247 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:50.172988 21025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } }
I20260620 04:39:50.173336 21025 leader_election.cc:302] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a3250e1c504c30b03b55e5f56e29c2; no voters:
I20260620 04:39:50.173695 21025 leader_election.cc:288] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:50.173764 21034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:50.173983 21034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [term 1 LEADER]: Becoming Leader. State: Replica: 93a3250e1c504c30b03b55e5f56e29c2, State: Running, Role: LEADER
I20260620 04:39:50.174314 21034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } }
I20260620 04:39:50.174911 21025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:50.175866 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93a3250e1c504c30b03b55e5f56e29c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } } }
I20260620 04:39:50.175989 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:50.176211 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a3250e1c504c30b03b55e5f56e29c2. Latest consensus state: current_term: 1 leader_uuid: "93a3250e1c504c30b03b55e5f56e29c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a3250e1c504c30b03b55e5f56e29c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36647 } } }
I20260620 04:39:50.176296 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:50.176659 21044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:50.179661 21044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:50.183458 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:50.174418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50241 (local address 127.0.0.1:36647)
0620 04:39:50.174535 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:50.174539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:50.174550 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:39:50.174588 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:50.174590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:50.174620 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:50.174705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:50.175442 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:50.176225 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:50.177526 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:50.177888 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:50.177975 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:50.179690 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:50.179694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:50.179696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:50.179720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:50.181544 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:50.181986 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:50.181991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:50.181993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:50.182079 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:50.182358 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:50.182362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:50.182364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:50.182538 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:50.182676 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:50.182904 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 04:39:50.183022 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:39:50.185523 21044 catalog_manager.cc:1347] Generated new cluster ID: f9cecf605ab04b898a1f1412217aba0e
I20260620 04:39:50.185608 21044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:50.206751 21044 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:50.207969 21044 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:50.222203 21044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a3250e1c504c30b03b55e5f56e29c2: Generated new TSK 0
I20260620 04:39:50.223059 21044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:50.238517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 20957
2026-06-20T04:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:50 dist-test-slave-zmf4 krb5kdc[21056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:50 dist-test-slave-zmf4 krb5kdc[21056](info): set up 2 sockets
Jun 20 04:39:50 dist-test-slave-zmf4 krb5kdc[21056](info): commencing operation
krb5kdc: starting...
W20260620 04:39:51.296226 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:51 dist-test-slave-zmf4 krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:51.415743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:51.525393 21069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:51.525664 21069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:51.525738 21069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:51.529716 21069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:51.529805 21069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:51.529844 21069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:51.529891 21069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:51.529922 21069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:51.535035 21069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:51.536055 21069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:51.537055 21069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:51.545132 21077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:51.545156 21074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:51.545205 21069 server_base.cc:1034] running on GCE node
W20260620 04:39:51.545156 21075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:51.545903 21069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:51.546921 21069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:51.548131 21069 hybrid_clock.cc:648] HybridClock initialized: now 1781930391548120 us; error 25 us; skew 500 ppm
Jun 20 04:39:51 dist-test-slave-zmf4 krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:51.555969 21069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:51.556915 21069 webserver.cc:460] Webserver started at http://127.0.0.1:38281/ using document root <none> and password file <none>
I20260620 04:39:51.557432 21069 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:51.557497 21069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:51.557669 21069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:51.559634 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "106ffff6748f46c3b1c895e02bf2578c"
format_stamp: "Formatted at 2026-06-20 04:39:51 on dist-test-slave-zmf4"
server_key: "1c72a27397f170d6304224ab49a08246"
server_key_iv: "c6e1eece16704d2ce9f88718757935d6"
server_key_version: "encryptionkey@0"
I20260620 04:39:51.560138 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "106ffff6748f46c3b1c895e02bf2578c"
format_stamp: "Formatted at 2026-06-20 04:39:51 on dist-test-slave-zmf4"
server_key: "1c72a27397f170d6304224ab49a08246"
server_key_iv: "c6e1eece16704d2ce9f88718757935d6"
server_key_version: "encryptionkey@0"
I20260620 04:39:51.563769 21069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 04:39:51.566205 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:51.567492 21069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:39:51.567644 21069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "106ffff6748f46c3b1c895e02bf2578c"
format_stamp: "Formatted at 2026-06-20 04:39:51 on dist-test-slave-zmf4"
server_key: "1c72a27397f170d6304224ab49a08246"
server_key_iv: "c6e1eece16704d2ce9f88718757935d6"
server_key_version: "encryptionkey@0"
I20260620 04:39:51.567752 21069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:51.589345 21069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:51.592907 21069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:51.593225 21069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:51.600771 21069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44515
I20260620 04:39:51.600950 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44515 every 8 connection(s)
I20260620 04:39:51.601766 21069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:51.604588 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:51.609855 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c: Bootstrap starting.
I20260620 04:39:51.611836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21069
I20260620 04:39:51.611985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:51.612294 392 external_mini_cluster.cc:1428] Setting key 36588859bddb5afc1a680e81638aa86c
I20260620 04:39:51.612491 21137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:51.613509 21137 log.cc:824] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:51.615680 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c: No bootstrap required, opened a new log
I20260620 04:39:51.619921 21137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 04:39:51.620136 21137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:51.620198 21137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106ffff6748f46c3b1c895e02bf2578c, State: Initialized, Role: FOLLOWER
I20260620 04:39:51.620649 21137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 04:39:51.620777 21137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:51.620889 21137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:51.621132 21137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:51.622206 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:51.613632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48213 (local address 127.0.0.1:44515)
0620 04:39:51.614092 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:51.614106 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:51.614140 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:39:51.614983 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:51.615006 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:51.615391 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:51.615750 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:51.616144 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:51.617674 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:51.618577 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:51.619319 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:51.619494 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:51.620412 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:51.620417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:51.620430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:51.620944 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:51.620997 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:51.621020 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:51.621165 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:51.621283 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 04:39:51.621488 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"spinlock_wait_cycles":24832,"thread_start_us":180,"threads_started":1}
I20260620 04:39:51.622833 21137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 04:39:51.623163 21137 leader_election.cc:302] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106ffff6748f46c3b1c895e02bf2578c; no voters:
I20260620 04:39:51.623466 21137 leader_election.cc:288] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:51.623570 21142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:51.623771 21142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [term 1 LEADER]: Becoming Leader. State: Replica: 106ffff6748f46c3b1c895e02bf2578c, State: Running, Role: LEADER
I20260620 04:39:51.624404 21142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 04:39:51.624794 21137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:51.625130 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:51.626056 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106ffff6748f46c3b1c895e02bf2578c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260620 04:39:51.626173 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:51.626478 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106ffff6748f46c3b1c895e02bf2578c. Latest consensus state: current_term: 1 leader_uuid: "106ffff6748f46c3b1c895e02bf2578c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106ffff6748f46c3b1c895e02bf2578c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260620 04:39:51.626549 21151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:51.626580 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:51.629781 21151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:51.632531 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:51.627512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39325 (local address 127.0.0.1:44515)
0620 04:39:51.627624 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:51.627628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:51.628689 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 04:39:51.628811 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:51.628814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:51.628851 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:51.628947 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:51.629190 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:51.629977 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:51.630898 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:51.631138 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:51.631243 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:51.632077 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:51.632081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:51.632084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:51.632097 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:51.632124 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:51.632137 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:51.632243 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:51.632342 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 04:39:51.632396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:39:51.635785 21151 catalog_manager.cc:1347] Generated new cluster ID: 891da8c2452148609c867ace6558e2d6
I20260620 04:39:51.635848 21151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:51.653469 21151 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:51.654649 21151 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:51.665923 21151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106ffff6748f46c3b1c895e02bf2578c: Generated new TSK 0
I20260620 04:39:51.666612 21151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:51.675133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21069
2026-06-20T04:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:51 dist-test-slave-zmf4 krb5kdc[21168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:51 dist-test-slave-zmf4 krb5kdc[21168](info): set up 2 sockets
Jun 20 04:39:51 dist-test-slave-zmf4 krb5kdc[21168](info): commencing operation
krb5kdc: starting...
W20260620 04:39:52.720791 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:52 dist-test-slave-zmf4 krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:52.856112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:52.970415 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:52.970676 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:52.970718 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:52.974923 21181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:52.975009 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:52.975030 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:52.975047 21181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:52.975063 21181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:52.980378 21181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:52.981415 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:52.982424 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:52.990962 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:52.990976 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:52.990976 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:52.991271 21181 server_base.cc:1034] running on GCE node
I20260620 04:39:52.991961 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:52.993093 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:52.994320 21181 hybrid_clock.cc:648] HybridClock initialized: now 1781930392994291 us; error 38 us; skew 500 ppm
Jun 20 04:39:52 dist-test-slave-zmf4 krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:52.997435 21181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:52.998265 21181 webserver.cc:460] Webserver started at http://127.0.0.1:33023/ using document root <none> and password file <none>
I20260620 04:39:52.998812 21181 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:52.998898 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:52.999114 21181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:53.001116 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "0db29d5cf5e54d3c9b389f7c80c7892b"
format_stamp: "Formatted at 2026-06-20 04:39:52 on dist-test-slave-zmf4"
server_key: "a11a27a51ee7437098c7685d18e9930f"
server_key_iv: "c1ccf0f9c205501375d3a923d4763367"
server_key_version: "encryptionkey@0"
I20260620 04:39:53.001667 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "0db29d5cf5e54d3c9b389f7c80c7892b"
format_stamp: "Formatted at 2026-06-20 04:39:52 on dist-test-slave-zmf4"
server_key: "a11a27a51ee7437098c7685d18e9930f"
server_key_iv: "c1ccf0f9c205501375d3a923d4763367"
server_key_version: "encryptionkey@0"
I20260620 04:39:53.005280 21181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:39:53.007694 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:53.008942 21181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:39:53.009078 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "0db29d5cf5e54d3c9b389f7c80c7892b"
format_stamp: "Formatted at 2026-06-20 04:39:52 on dist-test-slave-zmf4"
server_key: "a11a27a51ee7437098c7685d18e9930f"
server_key_iv: "c1ccf0f9c205501375d3a923d4763367"
server_key_version: "encryptionkey@0"
I20260620 04:39:53.009189 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:53.013587 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:53.016501 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:53.016685 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:53.023528 21181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41775
I20260620 04:39:53.023550 21248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41775 every 8 connection(s)
I20260620 04:39:53.024457 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:53.027118 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:53.032572 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: Bootstrap starting.
I20260620 04:39:53.033509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21181
I20260620 04:39:53.033666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:53.033984 392 external_mini_cluster.cc:1428] Setting key 8b300d8f34cd695ab2ed427732c3b925
I20260620 04:39:53.035410 21249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:53.036145 21249 log.cc:824] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:53.038153 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: No bootstrap required, opened a new log
I20260620 04:39:53.038853 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:53.035050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38549 (local address 127.0.0.1:41775)
0620 04:39:53.035573 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:53.035587 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:53.035621 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:39:53.036323 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:53.036350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:53.036620 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:53.036972 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:53.037725 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:53.037729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:53.037740 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:53.037750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:53.037799 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:53.037817 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:53.037931 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:53.038014 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:39:53.038165 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":135,"threads_started":1}
I20260620 04:39:53.041798 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:53.042296 21249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } }
I20260620 04:39:53.042500 21249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:53.042572 21249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db29d5cf5e54d3c9b389f7c80c7892b, State: Initialized, Role: FOLLOWER
I20260620 04:39:53.042996 21249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } }
I20260620 04:39:53.043112 21249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:53.043244 21249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:53.043349 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:53.044446 21249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } }
I20260620 04:39:53.044838 21249 leader_election.cc:302] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0db29d5cf5e54d3c9b389f7c80c7892b; no voters:
I20260620 04:39:53.045204 21249 leader_election.cc:288] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:53.045351 21259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:53.045583 21259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [term 1 LEADER]: Becoming Leader. State: Replica: 0db29d5cf5e54d3c9b389f7c80c7892b, State: Running, Role: LEADER
I20260620 04:39:53.045894 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:53.044114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48679 (local address 127.0.0.1:41775)
0620 04:39:53.044242 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:53.044246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:53.044458 (+ 212us) server_negotiation.cc:407] Connection header received
0620 04:39:53.044602 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:53.044606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:53.044647 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:53.044712 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:53.045463 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:53.045466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:53.045469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:53.045480 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:53.045505 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:53.045517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:53.045572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:53.045733 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:39:53.045786 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:39:53.045948 21259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } }
I20260620 04:39:53.046553 21249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:53.047741 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } } }
I20260620 04:39:53.047869 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:53.048122 21260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0db29d5cf5e54d3c9b389f7c80c7892b. Latest consensus state: current_term: 1 leader_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db29d5cf5e54d3c9b389f7c80c7892b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41775 } } }
I20260620 04:39:53.048381 21260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b [sys.catalog]: This master's current role is: LEADER
W20260620 04:39:53.051472 21274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:39:53.051538 21274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:39:53.051614 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:53.052469 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:53.058132 21268 catalog_manager.cc:1347] Generated new cluster ID: d28b9423f7f84f0ba7a6a2debb3046d8
I20260620 04:39:53.058203 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:53.066035 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:53.066382 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:53.072422 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0db29d5cf5e54d3c9b389f7c80c7892b: Generated new TSK 0
I20260620 04:39:53.073092 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:53.083504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21181
2026-06-20T04:39:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:53 dist-test-slave-zmf4 krb5kdc[21280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:53 dist-test-slave-zmf4 krb5kdc[21280](info): set up 2 sockets
Jun 20 04:39:53 dist-test-slave-zmf4 krb5kdc[21280](info): commencing operation
krb5kdc: starting...
W20260620 04:39:54.132756 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:54.272169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:54.392612 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:54.392892 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:54.392974 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:54.397045 21293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:54.397110 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:54.397131 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:54.397148 21293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:54.397164 21293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:54.402537 21293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:54.403656 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:54.404768 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:54.413131 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:54.413334 21293 server_base.cc:1034] running on GCE node
W20260620 04:39:54.413131 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:54.413127 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:54.414319 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:54.415395 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:54.416617 21293 hybrid_clock.cc:648] HybridClock initialized: now 1781930394416586 us; error 47 us; skew 500 ppm
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:54.420265 21293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:54.421159 21293 webserver.cc:460] Webserver started at http://127.0.0.1:34883/ using document root <none> and password file <none>
I20260620 04:39:54.421751 21293 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:54.421828 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:54.422048 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:54.424161 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "01101e50cf014b148d4188124360216d"
format_stamp: "Formatted at 2026-06-20 04:39:54 on dist-test-slave-zmf4"
server_key: "91389b694a7f0751b9a7a33d582f0c7c"
server_key_iv: "8f997ba96c0bfa064e46199c9aad7034"
server_key_version: "encryptionkey@0"
I20260620 04:39:54.424824 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "01101e50cf014b148d4188124360216d"
format_stamp: "Formatted at 2026-06-20 04:39:54 on dist-test-slave-zmf4"
server_key: "91389b694a7f0751b9a7a33d582f0c7c"
server_key_iv: "8f997ba96c0bfa064e46199c9aad7034"
server_key_version: "encryptionkey@0"
I20260620 04:39:54.429065 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:39:54.431856 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:54.433386 21293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:54.433529 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "01101e50cf014b148d4188124360216d"
format_stamp: "Formatted at 2026-06-20 04:39:54 on dist-test-slave-zmf4"
server_key: "91389b694a7f0751b9a7a33d582f0c7c"
server_key_iv: "8f997ba96c0bfa064e46199c9aad7034"
server_key_version: "encryptionkey@0"
I20260620 04:39:54.433645 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:54.449183 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:54.452548 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:54.452765 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:54.462299 21293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34961
I20260620 04:39:54.462440 21360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34961 every 8 connection(s)
I20260620 04:39:54.463352 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:54.466243 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:54.469774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21293
I20260620 04:39:54.469923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:54.470237 392 external_mini_cluster.cc:1428] Setting key bb12b14360552d7b938d891772052656
I20260620 04:39:54.473946 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d: Bootstrap starting.
I20260620 04:39:54.476888 21361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:54.477653 21361 log.cc:824] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:54.480274 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d: No bootstrap required, opened a new log
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:39:54.485080 21361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 04:39:54.485380 21361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:54.485452 21361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01101e50cf014b148d4188124360216d, State: Initialized, Role: FOLLOWER
I20260620 04:39:54.485930 21361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 04:39:54.486122 21361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:54.486189 21361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:54.486302 21361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:54.487529 21361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 04:39:54.487888 21361 leader_election.cc:302] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01101e50cf014b148d4188124360216d; no voters:
I20260620 04:39:54.488256 21361 leader_election.cc:288] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:54.488430 21366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:54.488647 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:54.471752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:60145 (local address 127.0.0.1:34961)
0620 04:39:54.472532 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:54.472551 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:54.472593 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:39:54.473496 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:54.473521 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:54.473884 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:54.474340 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:54.475342 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:54.476553 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:54.478328 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:54.479417 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:54.479609 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:54.482716 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:54.482722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:54.482736 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:54.482762 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:54.485119 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:54.485722 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:54.485729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:54.485736 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:54.485879 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:54.486305 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:54.486310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:54.486315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:54.486820 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:54.487040 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:54.487375 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 04:39:54.487617 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":146,"threads_started":1}
I20260620 04:39:54.488677 21366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [term 1 LEADER]: Becoming Leader. State: Replica: 01101e50cf014b148d4188124360216d, State: Running, Role: LEADER
I20260620 04:39:54.489023 21366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 04:39:54.489523 21361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:54.490144 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01101e50cf014b148d4188124360216d. Latest consensus state: current_term: 1 leader_uuid: "01101e50cf014b148d4188124360216d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260620 04:39:54.490329 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:54.490581 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01101e50cf014b148d4188124360216d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01101e50cf014b148d4188124360216d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260620 04:39:54.490684 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:54.490881 21375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:54.491873 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:54.494203 21375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:54.504814 21375 catalog_manager.cc:1347] Generated new cluster ID: 0d6dc746065e4a06a49cad8bf12555fc
I20260620 04:39:54.504922 21375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:54.509033 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:54.497029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:45179 (local address 127.0.0.1:34961)
0620 04:39:54.497142 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:54.497146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:54.498552 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 04:39:54.498742 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:54.498745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:54.498798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:54.498895 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:54.499578 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:54.500525 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:54.501895 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:54.502281 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:54.502439 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:54.504418 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:54.504422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:54.504424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:54.504455 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:54.506630 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:54.507587 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:54.507591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:54.507593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:54.507698 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:54.508077 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:54.508080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:54.508082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:54.508317 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:54.508446 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:54.508757 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 04:39:54.508856 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:39:54.517450 21375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:54.519001 21375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:54.528277 21375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01101e50cf014b148d4188124360216d: Generated new TSK 0
I20260620 04:39:54.529172 21375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:54.534832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21293
W20260620 04:39:54.540347 21278 connection.cc:535] client connection to 127.0.0.1:34961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34961): Connection reset by peer (error 104)
2026-06-20T04:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21392](info): set up 2 sockets
Jun 20 04:39:54 dist-test-slave-zmf4 krb5kdc[21392](info): commencing operation
krb5kdc: starting...
W20260620 04:39:55.616617 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:55 dist-test-slave-zmf4 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:55.743273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:55.857276 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:55.857590 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:55.857672 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:55.861724 21405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:55.861817 21405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:55.861855 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:55.861922 21405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:55.861958 21405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:55.867430 21405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:55.868552 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:55.869660 21405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:55.878106 21410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:55.878118 21413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:55.878130 21411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:55.878602 21405 server_base.cc:1034] running on GCE node
I20260620 04:39:55.879315 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:55.880492 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:55.881740 21405 hybrid_clock.cc:648] HybridClock initialized: now 1781930395881717 us; error 49 us; skew 500 ppm
Jun 20 04:39:55 dist-test-slave-zmf4 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:55.884927 21405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:55.885782 21405 webserver.cc:460] Webserver started at http://127.0.0.1:46793/ using document root <none> and password file <none>
I20260620 04:39:55.886500 21405 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:55.886590 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:55.886796 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:55.888876 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "c9260f092d39433d96896321d49a6718"
format_stamp: "Formatted at 2026-06-20 04:39:55 on dist-test-slave-zmf4"
server_key: "27f368fc18492801b20c44346743792e"
server_key_iv: "636ebe1c8ca9cd5d07208141e1accfe8"
server_key_version: "encryptionkey@0"
I20260620 04:39:55.889444 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9260f092d39433d96896321d49a6718"
format_stamp: "Formatted at 2026-06-20 04:39:55 on dist-test-slave-zmf4"
server_key: "27f368fc18492801b20c44346743792e"
server_key_iv: "636ebe1c8ca9cd5d07208141e1accfe8"
server_key_version: "encryptionkey@0"
I20260620 04:39:55.893332 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:39:55.895735 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:55.897492 21405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:55.897644 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "c9260f092d39433d96896321d49a6718"
format_stamp: "Formatted at 2026-06-20 04:39:55 on dist-test-slave-zmf4"
server_key: "27f368fc18492801b20c44346743792e"
server_key_iv: "636ebe1c8ca9cd5d07208141e1accfe8"
server_key_version: "encryptionkey@0"
I20260620 04:39:55.897769 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:55.919835 21405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:55.923365 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:55.923602 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:55.930997 21405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42411
I20260620 04:39:55.931018 21472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42411 every 8 connection(s)
I20260620 04:39:55.931922 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:55.934733 21473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:55.939996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21405
I20260620 04:39:55.940173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:55.940320 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718: Bootstrap starting.
I20260620 04:39:55.940572 392 external_mini_cluster.cc:1428] Setting key 0dd942d63263022b98266e1e4d695304
I20260620 04:39:55.942930 21473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:55.943662 21473 log.cc:824] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:55.945683 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718: No bootstrap required, opened a new log
I20260620 04:39:55.949880 21473 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260620 04:39:55.950062 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:55.941981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:53015 (local address 127.0.0.1:42411)
0620 04:39:55.942441 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:55.942458 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:55.942583 (+ 125us) server_negotiation.cc:407] Connection header received
0620 04:39:55.943475 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:55.943493 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:55.943992 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:55.944367 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:55.944734 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:55.945652 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:55.946830 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:55.947310 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:55.947432 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:55.948365 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:55.948370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:55.948383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:55.948754 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:55.948805 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:55.948825 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:55.948939 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:55.949213 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 04:39:55.949358 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260620 04:39:55.950088 21473 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:55.950253 21473 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9260f092d39433d96896321d49a6718, State: Initialized, Role: FOLLOWER
I20260620 04:39:55.950696 21473 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260620 04:39:55.950822 21473 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:55.950904 21473 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:55.951017 21473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:55.952107 21473 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260620 04:39:55.952521 21473 leader_election.cc:302] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9260f092d39433d96896321d49a6718; no voters:
I20260620 04:39:55.952872 21473 leader_election.cc:288] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:55.952934 21478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:55.953136 21478 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [term 1 LEADER]: Becoming Leader. State: Replica: c9260f092d39433d96896321d49a6718, State: Running, Role: LEADER
I20260620 04:39:55.953693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:55.953644 21478 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260620 04:39:55.954298 21473 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:55.954838 21478 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9260f092d39433d96896321d49a6718. Latest consensus state: current_term: 1 leader_uuid: "c9260f092d39433d96896321d49a6718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260620 04:39:55.955605 21478 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:55.955489 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9260f092d39433d96896321d49a6718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9260f092d39433d96896321d49a6718" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260620 04:39:55.955907 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:55.956584 21491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:55.959434 21491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:55.960741 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:55.955895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:60395 (local address 127.0.0.1:42411)
0620 04:39:55.955994 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:55.955998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:55.956454 (+ 456us) server_negotiation.cc:407] Connection header received
0620 04:39:55.956642 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:55.956646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:55.956693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:55.956793 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:55.957033 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:55.957771 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:55.958892 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:55.959206 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:55.959311 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:55.960221 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:55.960225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:55.960228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:55.960240 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:55.960265 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:55.960279 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:55.960364 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:55.960584 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 04:39:55.960620 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:39:55.965503 21491 catalog_manager.cc:1347] Generated new cluster ID: 96c6187f96d84957b41c73cecaf9a0d1
I20260620 04:39:55.965572 21491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:55.977044 21491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:55.978267 21491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:55.999061 21491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9260f092d39433d96896321d49a6718: Generated new TSK 0
I20260620 04:39:55.999936 21491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:56.010088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21405
W20260620 04:39:56.016086 21390 connection.cc:535] client connection to 127.0.0.1:42411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42411): Connection reset by peer (error 104)
2026-06-20T04:39:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:56 dist-test-slave-zmf4 krb5kdc[21504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:56 dist-test-slave-zmf4 krb5kdc[21504](info): set up 2 sockets
Jun 20 04:39:56 dist-test-slave-zmf4 krb5kdc[21504](info): commencing operation
krb5kdc: starting...
W20260620 04:39:57.072625 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:57 dist-test-slave-zmf4 krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:57.200773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:57.318986 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:57.319263 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:57.319306 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:57.324288 21517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:57.324357 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:57.324376 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:57.324394 21517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:57.324409 21517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:57.329823 21517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:57.331072 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:57.332183 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:39:57.341209 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:57.341209 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:57.341645 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:57.342540 21517 server_base.cc:1034] running on GCE node
I20260620 04:39:57.343123 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:57.344313 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:57.345496 21517 hybrid_clock.cc:648] HybridClock initialized: now 1781930397345470 us; error 41 us; skew 500 ppm
Jun 20 04:39:57 dist-test-slave-zmf4 krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:57.352075 21517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:57.353034 21517 webserver.cc:460] Webserver started at http://127.0.0.1:45811/ using document root <none> and password file <none>
I20260620 04:39:57.353619 21517 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:57.353709 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:57.353955 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:57.356019 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "0f9de90a324448d9a75fc248924349f2"
format_stamp: "Formatted at 2026-06-20 04:39:57 on dist-test-slave-zmf4"
server_key: "4433a22f492298d302d975065ea82e26"
server_key_iv: "2adc978670bdb2cc9a9d6e34681047ef"
server_key_version: "encryptionkey@0"
I20260620 04:39:57.356653 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f9de90a324448d9a75fc248924349f2"
format_stamp: "Formatted at 2026-06-20 04:39:57 on dist-test-slave-zmf4"
server_key: "4433a22f492298d302d975065ea82e26"
server_key_iv: "2adc978670bdb2cc9a9d6e34681047ef"
server_key_version: "encryptionkey@0"
I20260620 04:39:57.360402 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:39:57.362988 21532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:57.364645 21517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:39:57.364797 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "0f9de90a324448d9a75fc248924349f2"
format_stamp: "Formatted at 2026-06-20 04:39:57 on dist-test-slave-zmf4"
server_key: "4433a22f492298d302d975065ea82e26"
server_key_iv: "2adc978670bdb2cc9a9d6e34681047ef"
server_key_version: "encryptionkey@0"
I20260620 04:39:57.364923 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:57.369422 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:57.372870 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:57.373070 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:57.380893 21517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44221
I20260620 04:39:57.380918 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44221 every 8 connection(s)
I20260620 04:39:57.382030 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:57.384739 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:57.387490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21517
I20260620 04:39:57.387642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:57.387924 392 external_mini_cluster.cc:1428] Setting key 6e1988056308b2f928f35f2c7482040c
I20260620 04:39:57.390533 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2: Bootstrap starting.
I20260620 04:39:57.392844 21585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:57.392927 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:57.389325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:60579 (local address 127.0.0.1:44221)
0620 04:39:57.389778 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:57.389792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:57.389826 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:39:57.390414 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:57.390430 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:57.390742 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:57.391057 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:57.391796 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:57.391800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:57.391811 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:57.391821 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:57.391871 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:57.391890 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:57.392028 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:57.392095 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 04:39:57.392267 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":138,"threads_started":1}
I20260620 04:39:57.393581 21585 log.cc:824] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:39:57.395745 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2: No bootstrap required, opened a new log
I20260620 04:39:57.396116 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:57.399905 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:57.398122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:43105 (local address 127.0.0.1:44221)
0620 04:39:57.398254 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:57.398258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:57.398462 (+ 204us) server_negotiation.cc:407] Connection header received
0620 04:39:57.398639 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:57.398642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:57.398679 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:57.398759 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:57.399434 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:57.399438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:57.399440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:39:57.399451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:57.399476 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:39:57.399489 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:57.399581 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:57.399700 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 04:39:57.399736 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:39:57.399775 21585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260620 04:39:57.400095 21585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:57.400159 21585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9de90a324448d9a75fc248924349f2, State: Initialized, Role: FOLLOWER
I20260620 04:39:57.400593 21585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260620 04:39:57.400712 21585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:57.400799 21585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:57.400918 21585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:57.401784 21585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260620 04:39:57.402061 21585 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9de90a324448d9a75fc248924349f2; no voters:
I20260620 04:39:57.402289 21585 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:57.402367 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:57.402624 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9de90a324448d9a75fc248924349f2, State: Running, Role: LEADER
I20260620 04:39:57.402920 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260620 04:39:57.403821 21585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:57.404549 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9de90a324448d9a75fc248924349f2. Latest consensus state: current_term: 1 leader_uuid: "0f9de90a324448d9a75fc248924349f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } } }
I20260620 04:39:57.404527 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f9de90a324448d9a75fc248924349f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9de90a324448d9a75fc248924349f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } } }
I20260620 04:39:57.404670 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:57.404877 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:57.405001 21602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:57.407869 21602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:57.413347 21602 catalog_manager.cc:1347] Generated new cluster ID: 1ee1c3080aac4085a5b8600892a736e1
I20260620 04:39:57.413425 21602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:57.424722 21602 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:57.425025 21602 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:57.432389 21602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f9de90a324448d9a75fc248924349f2: Generated new TSK 0
I20260620 04:39:57.433136 21602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:57.443106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21517
2026-06-20T04:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:57 dist-test-slave-zmf4 krb5kdc[21616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:57 dist-test-slave-zmf4 krb5kdc[21616](info): set up 2 sockets
Jun 20 04:39:57 dist-test-slave-zmf4 krb5kdc[21616](info): commencing operation
krb5kdc: starting...
W20260620 04:39:58.508517 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:39:58.636545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:39:58.745621 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:39:58.745905 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:39:58.745993 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:39:58.749941 21629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:39:58.750018 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:39:58.750063 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:39:58.750121 21629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:39:58.750160 21629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:39:58.755293 21629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:39:58.756347 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:39:58.757342 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:39:58.765730 21629 server_base.cc:1034] running on GCE node
W20260620 04:39:58.765724 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:58.765722 21637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:39:58.765787 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:39:58.766536 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:39:58.767612 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:39:58.768824 21629 hybrid_clock.cc:648] HybridClock initialized: now 1781930398768802 us; error 44 us; skew 500 ppm
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:39:58.771566 21629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:39:58.772567 21629 webserver.cc:460] Webserver started at http://127.0.0.1:34313/ using document root <none> and password file <none>
I20260620 04:39:58.773110 21629 fs_manager.cc:359] Metadata directory not provided
I20260620 04:39:58.773196 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:39:58.773442 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:39:58.775408 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "04193bf924254af888e2e12447f3c7d8"
format_stamp: "Formatted at 2026-06-20 04:39:58 on dist-test-slave-zmf4"
server_key: "6b0deb481a057b10d1da145e49d14234"
server_key_iv: "a122e849af5c148a41eade6d0aeb9079"
server_key_version: "encryptionkey@0"
I20260620 04:39:58.775955 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "04193bf924254af888e2e12447f3c7d8"
format_stamp: "Formatted at 2026-06-20 04:39:58 on dist-test-slave-zmf4"
server_key: "6b0deb481a057b10d1da145e49d14234"
server_key_iv: "a122e849af5c148a41eade6d0aeb9079"
server_key_version: "encryptionkey@0"
I20260620 04:39:58.779670 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:39:58.782291 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:39:58.783649 21629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:39:58.783885 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "04193bf924254af888e2e12447f3c7d8"
format_stamp: "Formatted at 2026-06-20 04:39:58 on dist-test-slave-zmf4"
server_key: "6b0deb481a057b10d1da145e49d14234"
server_key_iv: "a122e849af5c148a41eade6d0aeb9079"
server_key_version: "encryptionkey@0"
I20260620 04:39:58.784008 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:39:58.797703 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:39:58.800801 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:39:58.801011 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:39:58.807917 21629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43769
I20260620 04:39:58.807926 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43769 every 8 connection(s)
I20260620 04:39:58.808849 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:39:58.811492 21697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:39:58.813333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21629
I20260620 04:39:58.813469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:39:58.813746 392 external_mini_cluster.cc:1428] Setting key 4127c162302f513afbf03e7463fb681e
I20260620 04:39:58.817961 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8: Bootstrap starting.
I20260620 04:39:58.820442 21697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8: Neither blocks nor log segments found. Creating new log.
I20260620 04:39:58.821331 21697 log.cc:824] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:39:58.823777 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8: No bootstrap required, opened a new log
I20260620 04:39:58.827198 21697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } }
I20260620 04:39:58.827437 21697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:39:58.827533 21697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04193bf924254af888e2e12447f3c7d8, State: Initialized, Role: FOLLOWER
I20260620 04:39:58.827988 21697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } }
I20260620 04:39:58.828118 21697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:39:58.828230 21697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:39:58.828352 21697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:39:58.828644 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:58.815087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:50211 (local address 127.0.0.1:43769)
0620 04:39:58.815472 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:58.815494 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:58.815527 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:39:58.816143 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:58.816159 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:58.816501 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:58.816768 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:58.817661 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:58.818703 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:58.819876 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:58.820367 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:58.820485 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:58.823733 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:58.823738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:58.823751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:58.823780 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:58.825538 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:58.826069 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:58.826075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:58.826079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:58.826183 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:58.826531 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:58.826534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:58.826536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:58.826908 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:58.827140 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:58.827455 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 04:39:58.827706 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260620 04:39:58.829319 21697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } }
I20260620 04:39:58.829666 21697 leader_election.cc:302] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04193bf924254af888e2e12447f3c7d8; no voters:
I20260620 04:39:58.830024 21697 leader_election.cc:288] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:39:58.830106 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:39:58.830293 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [term 1 LEADER]: Becoming Leader. State: Replica: 04193bf924254af888e2e12447f3c7d8, State: Running, Role: LEADER
I20260620 04:39:58.830642 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } }
I20260620 04:39:58.831291 21697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:39:58.832341 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04193bf924254af888e2e12447f3c7d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } } }
I20260620 04:39:58.832439 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:39:58.832482 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:58.832882 21711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:39:58.832844 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04193bf924254af888e2e12447f3c7d8. Latest consensus state: current_term: 1 leader_uuid: "04193bf924254af888e2e12447f3c7d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04193bf924254af888e2e12447f3c7d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43769 } } }
I20260620 04:39:58.833024 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:39:58.836297 21711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:39:58.843730 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:39:58.834873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:45167 (local address 127.0.0.1:43769)
0620 04:39:58.834967 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 04:39:58.834972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:39:58.835025 (+ 53us) server_negotiation.cc:407] Connection header received
0620 04:39:58.835171 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:39:58.835175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:39:58.835224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:39:58.835302 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:39:58.836086 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:58.836905 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:58.837709 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:39:58.837963 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:39:58.838073 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:39:58.839570 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:39:58.839574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:39:58.839576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:39:58.839601 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:39:58.841366 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:58.842010 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:58.842014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:58.842016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:58.842100 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:39:58.842433 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:39:58.842437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:39:58.842439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:39:58.842674 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:39:58.842816 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:39:58.843470 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 04:39:58.843566 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 04:39:58.844393 21711 catalog_manager.cc:1347] Generated new cluster ID: 5495240230694636af2e46c48a6a1587
I20260620 04:39:58.844458 21711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:39:58.856484 21711 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:39:58.857705 21711 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:39:58.866012 21711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04193bf924254af888e2e12447f3c7d8: Generated new TSK 0
I20260620 04:39:58.866829 21711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:39:58.875451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21629
W20260620 04:39:58.880187 21614 connection.cc:535] client connection to 127.0.0.1:43769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43769): Connection reset by peer (error 104)
2026-06-20T04:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21728](info): set up 2 sockets
Jun 20 04:39:58 dist-test-slave-zmf4 krb5kdc[21728](info): commencing operation
krb5kdc: starting...
W20260620 04:39:59.932500 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:00 dist-test-slave-zmf4 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:40:00.059373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:00.171376 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:00.171723 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:00.171816 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:00.176338 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:00.176435 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:00.176484 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:00.176523 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:00.176558 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:00.182307 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:00.183386 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:00.184410 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:00.192667 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:00.192669 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:00.192657 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:00.192747 21741 server_base.cc:1034] running on GCE node
I20260620 04:40:00.193408 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:00.194391 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:00.195614 21741 hybrid_clock.cc:648] HybridClock initialized: now 1781930400195588 us; error 31 us; skew 500 ppm
Jun 20 04:40:00 dist-test-slave-zmf4 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:00.198326 21741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:40:00.199182 21741 webserver.cc:460] Webserver started at http://127.0.0.1:37739/ using document root <none> and password file <none>
I20260620 04:40:00.199719 21741 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:00.199808 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:00.200031 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:00.202178 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "97f19e09a26549b3bec016192aef9970"
format_stamp: "Formatted at 2026-06-20 04:40:00 on dist-test-slave-zmf4"
server_key: "5b35c15b50fbe549f173af6e5cc43bee"
server_key_iv: "b74550ae9c841cf8b0f78f85fca6349d"
server_key_version: "encryptionkey@0"
I20260620 04:40:00.202755 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "97f19e09a26549b3bec016192aef9970"
format_stamp: "Formatted at 2026-06-20 04:40:00 on dist-test-slave-zmf4"
server_key: "5b35c15b50fbe549f173af6e5cc43bee"
server_key_iv: "b74550ae9c841cf8b0f78f85fca6349d"
server_key_version: "encryptionkey@0"
I20260620 04:40:00.206359 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 04:40:00.208763 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:00.210157 21741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:40:00.210321 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "97f19e09a26549b3bec016192aef9970"
format_stamp: "Formatted at 2026-06-20 04:40:00 on dist-test-slave-zmf4"
server_key: "5b35c15b50fbe549f173af6e5cc43bee"
server_key_iv: "b74550ae9c841cf8b0f78f85fca6349d"
server_key_version: "encryptionkey@0"
I20260620 04:40:00.210444 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:00.223541 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:00.226446 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:00.226642 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:00.233680 21741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46409
I20260620 04:40:00.233759 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46409 every 8 connection(s)
I20260620 04:40:00.234617 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:00.235721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21741
I20260620 04:40:00.235888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:00.236179 392 external_mini_cluster.cc:1428] Setting key 711feb717ad1cf63db59854476ee11c4
I20260620 04:40:00.237603 21809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:00.243168 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970: Bootstrap starting.
I20260620 04:40:00.245532 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:00.237564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:50351 (local address 127.0.0.1:46409)
0620 04:40:00.237955 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:00.237969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:00.238005 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:40:00.238616 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:00.238631 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:00.238944 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:00.239264 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:00.239639 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:00.240671 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:00.242049 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:00.242620 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:00.243088 (+ 468us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:00.243915 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:00.243920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:00.243932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:00.244362 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:00.244412 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:00.244434 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:00.244575 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:00.244737 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 04:40:00.244868 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20260620 04:40:00.245815 21809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:00.246659 21809 log.cc:824] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:00.248819 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:00.248960 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970: No bootstrap required, opened a new log
I20260620 04:40:00.253122 21809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } }
I20260620 04:40:00.253408 21809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:00.253459 21809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f19e09a26549b3bec016192aef9970, State: Initialized, Role: FOLLOWER
I20260620 04:40:00.253916 21809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } }
I20260620 04:40:00.254086 21809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:00.254158 21809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:00.254264 21809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:00.255036 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:00.250800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:40955 (local address 127.0.0.1:46409)
0620 04:40:00.250926 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:00.250931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:00.251142 (+ 211us) server_negotiation.cc:407] Connection header received
0620 04:40:00.251212 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:00.251216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:00.251270 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:00.251358 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:00.251642 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:00.252434 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:00.253270 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:00.253595 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:00.253701 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:00.254622 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:00.254626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:00.254629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:00.254642 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:00.254667 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:00.254679 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:00.254770 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:00.254871 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 04:40:00.254904 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:40:00.255470 21809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } }
I20260620 04:40:00.255821 21809 leader_election.cc:302] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97f19e09a26549b3bec016192aef9970; no voters:
I20260620 04:40:00.256143 21809 leader_election.cc:288] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:00.256264 21819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:00.256500 21819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [term 1 LEADER]: Becoming Leader. State: Replica: 97f19e09a26549b3bec016192aef9970, State: Running, Role: LEADER
I20260620 04:40:00.256851 21819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } }
I20260620 04:40:00.257385 21809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:00.258503 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97f19e09a26549b3bec016192aef9970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } } }
I20260620 04:40:00.258649 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:00.258965 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97f19e09a26549b3bec016192aef9970. Latest consensus state: current_term: 1 leader_uuid: "97f19e09a26549b3bec016192aef9970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f19e09a26549b3bec016192aef9970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46409 } } }
I20260620 04:40:00.259025 21828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:00.259115 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:00.263027 21828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:00.268970 21828 catalog_manager.cc:1347] Generated new cluster ID: 0c4fc3f35caf4be693f18088635b8dea
I20260620 04:40:00.269069 21828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:00.295220 21828 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:00.296303 21828 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:00.302207 21828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97f19e09a26549b3bec016192aef9970: Generated new TSK 0
I20260620 04:40:00.303165 21828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:00.314298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21741
W20260620 04:40:00.319080 21726 connection.cc:535] client connection to 127.0.0.1:46409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46409): Connection reset by peer (error 104)
2026-06-20T04:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:00 dist-test-slave-zmf4 krb5kdc[21840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:00 dist-test-slave-zmf4 krb5kdc[21840](info): set up 2 sockets
Jun 20 04:40:00 dist-test-slave-zmf4 krb5kdc[21840](info): commencing operation
krb5kdc: starting...
W20260620 04:40:01.368357 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:01 dist-test-slave-zmf4 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:40:01.487929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:01.603646 21853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:01.603935 21853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:01.604038 21853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:01.608207 21853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:01.608270 21853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:01.608290 21853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:01.608309 21853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:01.608323 21853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:01.613931 21853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:01.614996 21853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:01.615996 21853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:01.624895 21858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:01.624879 21859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:01.625140 21861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:01.625396 21853 server_base.cc:1034] running on GCE node
I20260620 04:40:01.625988 21853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:01.627118 21853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:01.628293 21853 hybrid_clock.cc:648] HybridClock initialized: now 1781930401628274 us; error 36 us; skew 500 ppm
Jun 20 04:40:01 dist-test-slave-zmf4 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:01.631325 21853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:40:01.632342 21853 webserver.cc:460] Webserver started at http://127.0.0.1:42885/ using document root <none> and password file <none>
I20260620 04:40:01.632928 21853 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:01.633020 21853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:01.633226 21853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:01.635380 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "4580244d2aba483e9b7411e61553cb90"
format_stamp: "Formatted at 2026-06-20 04:40:01 on dist-test-slave-zmf4"
server_key: "1f5f735dc9d957e79ecdb76690e7a5f7"
server_key_iv: "d0b171646996db79d6c8171f5b6f4d95"
server_key_version: "encryptionkey@0"
I20260620 04:40:01.635955 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "4580244d2aba483e9b7411e61553cb90"
format_stamp: "Formatted at 2026-06-20 04:40:01 on dist-test-slave-zmf4"
server_key: "1f5f735dc9d957e79ecdb76690e7a5f7"
server_key_iv: "d0b171646996db79d6c8171f5b6f4d95"
server_key_version: "encryptionkey@0"
I20260620 04:40:01.641113 21853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 04:40:01.643877 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:01.645181 21853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:40:01.645332 21853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "4580244d2aba483e9b7411e61553cb90"
format_stamp: "Formatted at 2026-06-20 04:40:01 on dist-test-slave-zmf4"
server_key: "1f5f735dc9d957e79ecdb76690e7a5f7"
server_key_iv: "d0b171646996db79d6c8171f5b6f4d95"
server_key_version: "encryptionkey@0"
I20260620 04:40:01.645452 21853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:01.649998 21853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:01.653177 21853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:01.653487 21853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:01.662259 21853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46393
I20260620 04:40:01.662494 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46393 every 8 connection(s)
I20260620 04:40:01.663281 21853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:01.665027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21853
I20260620 04:40:01.665202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:01.665503 392 external_mini_cluster.cc:1428] Setting key 35755977e3f37dcdb4e79d4cbacd8fdd
I20260620 04:40:01.666332 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:01.671033 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:01.666831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:36765 (local address 127.0.0.1:46393)
0620 04:40:01.667227 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:01.667241 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:01.667280 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:40:01.667895 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:01.667910 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:01.668170 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:01.668505 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:01.669504 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:01.669511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:01.669531 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:01.669551 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:01.669657 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:01.669687 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:01.669876 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:01.670007 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 04:40:01.670361 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":132,"threads_started":1}
I20260620 04:40:01.673066 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90: Bootstrap starting.
I20260620 04:40:01.674152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:01.675479 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:01.676290 21921 log.cc:824] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:01.678201 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90: No bootstrap required, opened a new log
I20260620 04:40:01.678246 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:01.676237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:55065 (local address 127.0.0.1:46393)
0620 04:40:01.676365 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:01.676371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:01.676684 (+ 313us) server_negotiation.cc:407] Connection header received
0620 04:40:01.676838 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:01.676842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:01.676882 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:01.676977 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:01.677802 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:01.677806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:01.677809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:01.677821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:01.677852 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:01.677881 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:01.677985 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:01.678066 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 04:40:01.678111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:40:01.681869 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260620 04:40:01.682086 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:01.682121 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4580244d2aba483e9b7411e61553cb90, State: Initialized, Role: FOLLOWER
I20260620 04:40:01.682575 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260620 04:40:01.682695 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:01.682760 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:01.682857 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:01.683876 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260620 04:40:01.684286 21921 leader_election.cc:302] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4580244d2aba483e9b7411e61553cb90; no voters:
I20260620 04:40:01.684600 21921 leader_election.cc:288] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:01.684717 21931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:01.684942 21931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [term 1 LEADER]: Becoming Leader. State: Replica: 4580244d2aba483e9b7411e61553cb90, State: Running, Role: LEADER
I20260620 04:40:01.685271 21931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260620 04:40:01.686028 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:01.686895 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4580244d2aba483e9b7411e61553cb90. Latest consensus state: current_term: 1 leader_uuid: "4580244d2aba483e9b7411e61553cb90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260620 04:40:01.687109 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4580244d2aba483e9b7411e61553cb90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4580244d2aba483e9b7411e61553cb90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260620 04:40:01.687218 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:01.687388 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:01.688066 21941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:01.690893 21941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:01.696009 21941 catalog_manager.cc:1347] Generated new cluster ID: b13bcdeb31eb47c9b09ddc35da985e65
I20260620 04:40:01.696080 21941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:01.724316 21941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:01.724673 21941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:01.733237 21941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4580244d2aba483e9b7411e61553cb90: Generated new TSK 0
I20260620 04:40:01.733913 21941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:01.745929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21853
2026-06-20T04:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:01 dist-test-slave-zmf4 krb5kdc[21952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:01 dist-test-slave-zmf4 krb5kdc[21952](info): set up 2 sockets
Jun 20 04:40:01 dist-test-slave-zmf4 krb5kdc[21952](info): commencing operation
krb5kdc: starting...
W20260620 04:40:02.804417 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:02 dist-test-slave-zmf4 krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:02.931491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:03.038537 21965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:03.038776 21965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:03.038815 21965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:03.042646 21965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:03.042701 21965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:03.042721 21965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:03.042737 21965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:03.042752 21965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:03.047674 21965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:03.048583 21965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:03.049487 21965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:03.057675 21971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:03.057660 21970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:03.057660 21973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:03.057933 21965 server_base.cc:1034] running on GCE node
I20260620 04:40:03.058496 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:03.059552 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:03.060796 21965 hybrid_clock.cc:648] HybridClock initialized: now 1781930403060778 us; error 33 us; skew 500 ppm
Jun 20 04:40:03 dist-test-slave-zmf4 krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:03.068001 21965 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:03.068919 21965 webserver.cc:460] Webserver started at http://127.0.98.62:34487/ using document root <none> and password file <none>
I20260620 04:40:03.069425 21965 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:03.069481 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:03.069636 21965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:03.071533 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "8df328efc8aa432eaf42464697f982bd"
format_stamp: "Formatted at 2026-06-20 04:40:03 on dist-test-slave-zmf4"
server_key: "8cbce91738ada639b880061c40ca2330"
server_key_iv: "99507ee57b433967bc794e79d86c2e08"
server_key_version: "encryptionkey@0"
I20260620 04:40:03.072031 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "8df328efc8aa432eaf42464697f982bd"
format_stamp: "Formatted at 2026-06-20 04:40:03 on dist-test-slave-zmf4"
server_key: "8cbce91738ada639b880061c40ca2330"
server_key_iv: "99507ee57b433967bc794e79d86c2e08"
server_key_version: "encryptionkey@0"
I20260620 04:40:03.075525 21965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:40:03.078038 21980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:03.079275 21965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:40:03.079449 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "8df328efc8aa432eaf42464697f982bd"
format_stamp: "Formatted at 2026-06-20 04:40:03 on dist-test-slave-zmf4"
server_key: "8cbce91738ada639b880061c40ca2330"
server_key_iv: "99507ee57b433967bc794e79d86c2e08"
server_key_version: "encryptionkey@0"
I20260620 04:40:03.079558 21965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:03.096838 21965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:03.100085 21965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:03.100333 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:03.107949 21965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43623
I20260620 04:40:03.108014 22032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43623 every 8 connection(s)
I20260620 04:40:03.108928 21965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:03.111704 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:03.117156 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: Bootstrap starting.
I20260620 04:40:03.117885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 21965
I20260620 04:40:03.118057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:03.118350 392 external_mini_cluster.cc:1428] Setting key a696c33d12878c1392aa2c366ae0091a
I20260620 04:40:03.119762 22033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:03.120556 22033 log.cc:824] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:03.122362 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: No bootstrap required, opened a new log
I20260620 04:40:03.125890 22033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } }
I20260620 04:40:03.126127 22033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:03.126220 22033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df328efc8aa432eaf42464697f982bd, State: Initialized, Role: FOLLOWER
I20260620 04:40:03.126660 22033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } }
I20260620 04:40:03.126837 22033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:03.126902 22033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:03.126999 22033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:40:03 dist-test-slave-zmf4 krb5kdc[21952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:03.127985 22033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } }
I20260620 04:40:03.128400 22033 leader_election.cc:302] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df328efc8aa432eaf42464697f982bd; no voters:
I20260620 04:40:03.128777 22033 leader_election.cc:288] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:03.128842 22038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:03.129031 22038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [term 1 LEADER]: Becoming Leader. State: Replica: 8df328efc8aa432eaf42464697f982bd, State: Running, Role: LEADER
I20260620 04:40:03.129343 22038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } }
I20260620 04:40:03.130040 22033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:03.130483 22038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df328efc8aa432eaf42464697f982bd. Latest consensus state: current_term: 1 leader_uuid: "8df328efc8aa432eaf42464697f982bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } } }
I20260620 04:40:03.130609 22038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:03.130476 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df328efc8aa432eaf42464697f982bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df328efc8aa432eaf42464697f982bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43623 } } }
I20260620 04:40:03.130962 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:03.133942 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:03.119458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54295 (local address 127.0.98.62:43623)
0620 04:40:03.120007 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:03.120023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:03.120059 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:40:03.120803 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:03.120820 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:03.121136 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:03.121475 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:03.122293 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:03.123291 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:03.124164 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:03.124698 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:03.124883 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:03.127974 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:03.127979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:03.127992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:03.128019 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:03.130182 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:03.130765 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:03.130771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:03.130779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:03.130931 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:03.131251 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:03.131256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:03.131257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:03.131759 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:03.131964 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:03.132740 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 04:40:03.132988 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
W20260620 04:40:03.134852 22053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:03.134927 22053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:03.134989 22043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:03.135820 22043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:03.137324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:03.141476 22043 catalog_manager.cc:1347] Generated new cluster ID: a2144a5e54a6451f87a97dbfef393665
I20260620 04:40:03.141541 22043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:03.148371 22043 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:03.149324 22043 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:03.151466 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:03.139446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45027 (local address 127.0.98.62:43623)
0620 04:40:03.139554 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:03.139558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:03.139809 (+ 251us) server_negotiation.cc:407] Connection header received
0620 04:40:03.139909 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:03.139912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:03.139958 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:03.140039 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:03.140953 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:03.144622 (+ 3669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:03.145782 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:03.146134 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:03.146229 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:03.147873 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:03.147877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:03.147880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:03.147913 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:03.149682 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:03.150209 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:03.150212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:03.150214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:03.150308 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:03.150630 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:03.150633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:03.150636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:03.150828 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:03.150963 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:03.151211 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 04:40:03.151315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:40:03.155959 22043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df328efc8aa432eaf42464697f982bd: Generated new TSK 0
I20260620 04:40:03.156754 22043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:03.167053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 21965
W20260620 04:40:03.172155 21950 connection.cc:535] client connection to 127.0.98.62:43623 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43623): Connection reset by peer (error 104)
2026-06-20T04:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:03 dist-test-slave-zmf4 krb5kdc[22064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:03 dist-test-slave-zmf4 krb5kdc[22064](info): set up 2 sockets
Jun 20 04:40:03 dist-test-slave-zmf4 krb5kdc[22064](info): commencing operation
krb5kdc: starting...
W20260620 04:40:04.228534 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:04 dist-test-slave-zmf4 krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:04.360991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:04.480372 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:04.480641 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:04.480686 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:04.485024 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:04.485092 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:04.485113 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:04.485131 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:04.485179 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:04.490875 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:04.492028 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:04.493081 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:04.501972 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:04.502040 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:04.501972 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:04.502360 22077 server_base.cc:1034] running on GCE node
I20260620 04:40:04.502928 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:04.503993 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:04.505175 22077 hybrid_clock.cc:648] HybridClock initialized: now 1781930404505151 us; error 42 us; skew 500 ppm
Jun 20 04:40:04 dist-test-slave-zmf4 krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:04.508045 22077 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:04.509008 22077 webserver.cc:460] Webserver started at http://127.0.98.62:41543/ using document root <none> and password file <none>
I20260620 04:40:04.509531 22077 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:04.509591 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:04.509753 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:04.511719 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "2bbfbbe028bd40878d2c30e26362234f"
format_stamp: "Formatted at 2026-06-20 04:40:04 on dist-test-slave-zmf4"
server_key: "a4df69baa622f4b31669c4fe80ddcc08"
server_key_iv: "581b4e95eccb57e40e660cac6d7a39ff"
server_key_version: "encryptionkey@0"
I20260620 04:40:04.512312 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "2bbfbbe028bd40878d2c30e26362234f"
format_stamp: "Formatted at 2026-06-20 04:40:04 on dist-test-slave-zmf4"
server_key: "a4df69baa622f4b31669c4fe80ddcc08"
server_key_iv: "581b4e95eccb57e40e660cac6d7a39ff"
server_key_version: "encryptionkey@0"
I20260620 04:40:04.516292 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:40:04.519014 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:04.520809 22077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:40:04.520922 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "2bbfbbe028bd40878d2c30e26362234f"
format_stamp: "Formatted at 2026-06-20 04:40:04 on dist-test-slave-zmf4"
server_key: "a4df69baa622f4b31669c4fe80ddcc08"
server_key_iv: "581b4e95eccb57e40e660cac6d7a39ff"
server_key_version: "encryptionkey@0"
I20260620 04:40:04.521070 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:04.536999 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:04.541482 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:04.541788 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:04.550820 22077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36255
I20260620 04:40:04.550885 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36255 every 8 connection(s)
I20260620 04:40:04.551925 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:04.554713 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:04.557790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22077
I20260620 04:40:04.557960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:04.558238 392 external_mini_cluster.cc:1428] Setting key 8ef543908c08de993c43eed4aaf7e622
I20260620 04:40:04.561010 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f: Bootstrap starting.
I20260620 04:40:04.563200 22145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:04.563944 22145 log.cc:824] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:04.565837 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f: No bootstrap required, opened a new log
I20260620 04:40:04.568135 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:04.559780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57217 (local address 127.0.98.62:36255)
0620 04:40:04.560527 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:04.560560 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:04.560631 (+ 71us) server_negotiation.cc:407] Connection header received
0620 04:40:04.561704 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:04.561728 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:04.562112 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:04.562481 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:04.562913 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:04.563953 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:04.564956 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:04.565424 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:04.565605 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:04.566513 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:04.566519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:04.566539 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:04.566926 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:04.566978 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:04.567001 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:04.567145 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:04.567292 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 04:40:04.567433 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":153,"threads_started":1}
I20260620 04:40:04.569581 22145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } }
I20260620 04:40:04.569837 22145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:04.569921 22145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbfbbe028bd40878d2c30e26362234f, State: Initialized, Role: FOLLOWER
I20260620 04:40:04.570395 22145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } }
I20260620 04:40:04.570573 22145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:04.570636 22145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:04.570735 22145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:04.571379 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:04.571862 22145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } }
I20260620 04:40:04.572247 22145 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bbfbbe028bd40878d2c30e26362234f; no voters:
I20260620 04:40:04.572594 22145 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:04.572664 22150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:04.572889 22150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [term 1 LEADER]: Becoming Leader. State: Replica: 2bbfbbe028bd40878d2c30e26362234f, State: Running, Role: LEADER
I20260620 04:40:04.573242 22150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } }
I20260620 04:40:04.574031 22145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:04.575224 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bbfbbe028bd40878d2c30e26362234f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } } }
I20260620 04:40:04.575362 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:04.575568 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbfbbe028bd40878d2c30e26362234f. Latest consensus state: current_term: 1 leader_uuid: "2bbfbbe028bd40878d2c30e26362234f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbfbbe028bd40878d2c30e26362234f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36255 } } }
I20260620 04:40:04.575639 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:04.576040 22164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:04.579559 22164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:04.579928 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:04.573781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47759 (local address 127.0.98.62:36255)
0620 04:40:04.573911 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:04.573916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:04.574430 (+ 514us) server_negotiation.cc:407] Connection header received
0620 04:40:04.574775 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:04.574778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:04.574827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:04.574886 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:04.575313 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:04.576083 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:04.577577 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:04.577905 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:04.578049 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:04.579333 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:04.579338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:04.579341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:04.579357 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:04.579395 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:04.579412 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:04.579517 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:04.579755 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 04:40:04.579797 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:40:04.588896 22164 catalog_manager.cc:1347] Generated new cluster ID: 7ef8e6f02503489eb3ece12205448ec3
I20260620 04:40:04.588986 22164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:04.607908 22164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:04.608949 22164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:04.615411 22164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbfbbe028bd40878d2c30e26362234f: Generated new TSK 0
I20260620 04:40:04.616293 22164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:04.626735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22077
W20260620 04:40:04.632462 22062 connection.cc:535] client connection to 127.0.98.62:36255 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36255): Connection reset by peer (error 104)
2026-06-20T04:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:04 dist-test-slave-zmf4 krb5kdc[22176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:04 dist-test-slave-zmf4 krb5kdc[22176](info): set up 2 sockets
Jun 20 04:40:04 dist-test-slave-zmf4 krb5kdc[22176](info): commencing operation
krb5kdc: starting...
W20260620 04:40:05.692911 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:05 dist-test-slave-zmf4 krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:05.831666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:05.952085 22189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:05.952464 22189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:05.952550 22189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:05.957669 22189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:05.957813 22189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:05.957856 22189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:05.957891 22189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:05.957923 22189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:05.963495 22189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:05.964723 22189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:05.965895 22189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:05.975075 22197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:05.975279 22189 server_base.cc:1034] running on GCE node
W20260620 04:40:05.975107 22194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:05.975085 22195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:05.975932 22189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:05.977156 22189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:05.978415 22189 hybrid_clock.cc:648] HybridClock initialized: now 1781930405978355 us; error 43 us; skew 500 ppm
Jun 20 04:40:05 dist-test-slave-zmf4 krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:05.981201 22189 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:05.982059 22189 webserver.cc:460] Webserver started at http://127.0.98.62:40579/ using document root <none> and password file <none>
I20260620 04:40:05.982625 22189 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:05.982717 22189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:05.982941 22189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:05.985062 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "9e4132bec9df4b968e97933f18ae9867"
format_stamp: "Formatted at 2026-06-20 04:40:05 on dist-test-slave-zmf4"
server_key: "04e5d1d6ff033548359523fc4a8251b2"
server_key_iv: "f4f66316feaa73b71159171701b4703b"
server_key_version: "encryptionkey@0"
I20260620 04:40:05.985725 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e4132bec9df4b968e97933f18ae9867"
format_stamp: "Formatted at 2026-06-20 04:40:05 on dist-test-slave-zmf4"
server_key: "04e5d1d6ff033548359523fc4a8251b2"
server_key_iv: "f4f66316feaa73b71159171701b4703b"
server_key_version: "encryptionkey@0"
I20260620 04:40:05.989977 22189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:40:05.993078 22204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:05.994921 22189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:40:05.995245 22189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "9e4132bec9df4b968e97933f18ae9867"
format_stamp: "Formatted at 2026-06-20 04:40:05 on dist-test-slave-zmf4"
server_key: "04e5d1d6ff033548359523fc4a8251b2"
server_key_iv: "f4f66316feaa73b71159171701b4703b"
server_key_version: "encryptionkey@0"
I20260620 04:40:05.995425 22189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:06.000622 22189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:06.004778 22189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:06.005012 22189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:06.013077 22189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42821
I20260620 04:40:06.013105 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42821 every 8 connection(s)
I20260620 04:40:06.014128 22189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:06.017156 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:06.019268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22189
I20260620 04:40:06.019409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:06.019683 392 external_mini_cluster.cc:1428] Setting key 2ecffbfcd5291f621fbf09d660a87b98
I20260620 04:40:06.023399 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867: Bootstrap starting.
I20260620 04:40:06.025225 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:06.021211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44077 (local address 127.0.98.62:42821)
0620 04:40:06.021676 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:06.021691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:06.021728 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:40:06.022579 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:06.022597 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:06.022894 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:06.023252 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:06.024030 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:06.024040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:06.024052 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:06.024061 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:06.024110 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:06.024130 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:06.024316 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:06.024388 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:40:06.024539 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260620 04:40:06.025897 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:06.026998 22257 log.cc:824] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:06.028498 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:06.029392 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867: No bootstrap required, opened a new log
I20260620 04:40:06.032516 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:06.030609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56615 (local address 127.0.98.62:42821)
0620 04:40:06.030734 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:06.030742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:06.030916 (+ 174us) server_negotiation.cc:407] Connection header received
0620 04:40:06.031063 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:06.031066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:06.031106 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:06.031195 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:06.032082 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:06.032086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:06.032088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:06.032099 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:06.032132 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:06.032148 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:06.032265 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:06.032338 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:40:06.032378 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:40:06.034150 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } }
I20260620 04:40:06.034404 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:06.034479 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4132bec9df4b968e97933f18ae9867, State: Initialized, Role: FOLLOWER
I20260620 04:40:06.034970 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } }
I20260620 04:40:06.035130 22257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:06.035207 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:06.035337 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:06.036621 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } }
I20260620 04:40:06.037004 22257 leader_election.cc:302] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e4132bec9df4b968e97933f18ae9867; no voters:
I20260620 04:40:06.037320 22257 leader_election.cc:288] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:06.037418 22267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:06.037587 22267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [term 1 LEADER]: Becoming Leader. State: Replica: 9e4132bec9df4b968e97933f18ae9867, State: Running, Role: LEADER
I20260620 04:40:06.037895 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } }
I20260620 04:40:06.038659 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:06.039687 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e4132bec9df4b968e97933f18ae9867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } } }
I20260620 04:40:06.039732 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e4132bec9df4b968e97933f18ae9867. Latest consensus state: current_term: 1 leader_uuid: "9e4132bec9df4b968e97933f18ae9867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4132bec9df4b968e97933f18ae9867" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42821 } } }
I20260620 04:40:06.039894 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:06.039992 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:06.040450 22276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:06.043325 22276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:06.049108 22276 catalog_manager.cc:1347] Generated new cluster ID: 8a73758164644adb9b5b24ece4ac6d7e
I20260620 04:40:06.049189 22276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:06.080770 22276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:06.081127 22276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:06.090474 22276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e4132bec9df4b968e97933f18ae9867: Generated new TSK 0
I20260620 04:40:06.091220 22276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:06.099511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22189
2026-06-20T04:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:06 dist-test-slave-zmf4 krb5kdc[22288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:06 dist-test-slave-zmf4 krb5kdc[22288](info): set up 2 sockets
Jun 20 04:40:06 dist-test-slave-zmf4 krb5kdc[22288](info): commencing operation
krb5kdc: starting...
W20260620 04:40:07.168835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:07.300386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:07.422912 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:07.423170 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:07.423213 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:07.427150 22301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:07.427209 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:07.427229 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:07.427245 22301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:07.427260 22301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:07.432505 22301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:07.433524 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:07.434495 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:07.442920 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:07.442930 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:07.442934 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:07.443272 22301 server_base.cc:1034] running on GCE node
I20260620 04:40:07.443899 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:07.445077 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:07.446316 22301 hybrid_clock.cc:648] HybridClock initialized: now 1781930407446278 us; error 49 us; skew 500 ppm
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:07.449257 22301 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:07.450198 22301 webserver.cc:460] Webserver started at http://127.0.98.62:35801/ using document root <none> and password file <none>
I20260620 04:40:07.450829 22301 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:07.450958 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:07.451197 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:07.453461 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "c16386752be144bc89cfab132f70705a"
format_stamp: "Formatted at 2026-06-20 04:40:07 on dist-test-slave-zmf4"
server_key: "f60d2a827cf00352e57354e379252d62"
server_key_iv: "5498e9a468ad0b49da66defee4e4a223"
server_key_version: "encryptionkey@0"
I20260620 04:40:07.454191 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c16386752be144bc89cfab132f70705a"
format_stamp: "Formatted at 2026-06-20 04:40:07 on dist-test-slave-zmf4"
server_key: "f60d2a827cf00352e57354e379252d62"
server_key_iv: "5498e9a468ad0b49da66defee4e4a223"
server_key_version: "encryptionkey@0"
I20260620 04:40:07.458292 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:40:07.460845 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:07.462146 22301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:40:07.462301 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "c16386752be144bc89cfab132f70705a"
format_stamp: "Formatted at 2026-06-20 04:40:07 on dist-test-slave-zmf4"
server_key: "f60d2a827cf00352e57354e379252d62"
server_key_iv: "5498e9a468ad0b49da66defee4e4a223"
server_key_version: "encryptionkey@0"
I20260620 04:40:07.462430 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:07.486243 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:07.489589 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:07.489816 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:07.497737 22301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37239
I20260620 04:40:07.497761 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37239 every 8 connection(s)
I20260620 04:40:07.498710 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:07.499315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22301
I20260620 04:40:07.499476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:07.499745 392 external_mini_cluster.cc:1428] Setting key dc2700a856da2978cf597ec9530f0748
I20260620 04:40:07.501596 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:07.507990 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a: Bootstrap starting.
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:07.510705 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:07.511626 22369 log.cc:824] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:07.513945 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a: No bootstrap required, opened a new log
I20260620 04:40:07.516130 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:07.501171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48973 (local address 127.0.98.62:37239)
0620 04:40:07.501586 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:07.501602 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:07.501658 (+ 56us) server_negotiation.cc:407] Connection header received
0620 04:40:07.502308 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:07.502325 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:07.502663 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:07.502958 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:07.503902 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:07.505025 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:07.506302 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:07.507122 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:07.507343 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:07.510199 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:07.510206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:07.510222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:07.510261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:07.512339 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:07.512782 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:07.512788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:07.512793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:07.512922 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:07.513242 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:07.513245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:07.513247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:07.513754 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:07.513954 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:07.514409 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 04:40:07.514825 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260620 04:40:07.519174 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:40:07.519428 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:07.519469 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16386752be144bc89cfab132f70705a, State: Initialized, Role: FOLLOWER
I20260620 04:40:07.520013 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:40:07.520164 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:07.520249 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:07.520313 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:07.520416 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:07.521792 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:40:07.522202 22369 leader_election.cc:302] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16386752be144bc89cfab132f70705a; no voters:
I20260620 04:40:07.522619 22369 leader_election.cc:288] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:07.522737 22378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:07.523006 22378 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [term 1 LEADER]: Becoming Leader. State: Replica: c16386752be144bc89cfab132f70705a, State: Running, Role: LEADER
I20260620 04:40:07.523376 22378 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:40:07.524165 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:07.524959 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c16386752be144bc89cfab132f70705a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:40:07.525074 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:07.525465 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16386752be144bc89cfab132f70705a. Latest consensus state: current_term: 1 leader_uuid: "c16386752be144bc89cfab132f70705a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16386752be144bc89cfab132f70705a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:40:07.525609 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:07.525906 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:07.529124 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:07.533303 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:07.522718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47249 (local address 127.0.98.62:37239)
0620 04:40:07.522859 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:07.522866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:07.523216 (+ 350us) server_negotiation.cc:407] Connection header received
0620 04:40:07.523428 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:07.523432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:07.523483 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:07.523540 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:07.524648 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:07.525565 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:07.526933 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:07.527321 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:07.527443 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:07.529442 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:07.529446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:07.529449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:07.529480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:07.531192 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:07.531723 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:07.531727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:07.531729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:07.531823 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:07.532139 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:07.532142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:07.532144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:07.532417 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:07.532696 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:07.533022 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 04:40:07.533149 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:40:07.537823 22386 catalog_manager.cc:1347] Generated new cluster ID: 86379a24f82d4130a1d23dc68efe39a8
I20260620 04:40:07.537911 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:07.563552 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:07.564790 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:07.571264 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c16386752be144bc89cfab132f70705a: Generated new TSK 0
I20260620 04:40:07.571970 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:07.578002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22301
2026-06-20T04:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22400](info): set up 2 sockets
Jun 20 04:40:07 dist-test-slave-zmf4 krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260620 04:40:08.648569 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:08 dist-test-slave-zmf4 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:08.776001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:08.883826 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:08.884084 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:08.884155 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:08.888487 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:08.888547 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:08.888569 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:08.888586 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:08.888638 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:08.894024 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:08.895061 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:08.896178 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:08.905179 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:08.905193 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:08.905180 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:08.905619 22413 server_base.cc:1034] running on GCE node
I20260620 04:40:08.906221 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:08.907305 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:08.908455 22413 hybrid_clock.cc:648] HybridClock initialized: now 1781930408908427 us; error 42 us; skew 500 ppm
Jun 20 04:40:08 dist-test-slave-zmf4 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:08.915230 22413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:08.916275 22413 webserver.cc:460] Webserver started at http://127.0.98.62:34887/ using document root <none> and password file <none>
I20260620 04:40:08.916891 22413 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:08.916986 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:08.917219 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:08.919225 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "caac826ddfcf4316b898ff39acb53c61"
format_stamp: "Formatted at 2026-06-20 04:40:08 on dist-test-slave-zmf4"
server_key: "2088d77c9d81387b044481752095322c"
server_key_iv: "d7939d14eb15f5cba74e2bdf6f677bbc"
server_key_version: "encryptionkey@0"
I20260620 04:40:08.919808 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "caac826ddfcf4316b898ff39acb53c61"
format_stamp: "Formatted at 2026-06-20 04:40:08 on dist-test-slave-zmf4"
server_key: "2088d77c9d81387b044481752095322c"
server_key_iv: "d7939d14eb15f5cba74e2bdf6f677bbc"
server_key_version: "encryptionkey@0"
I20260620 04:40:08.923527 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:40:08.926038 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:08.927624 22413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:40:08.927783 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "caac826ddfcf4316b898ff39acb53c61"
format_stamp: "Formatted at 2026-06-20 04:40:08 on dist-test-slave-zmf4"
server_key: "2088d77c9d81387b044481752095322c"
server_key_iv: "d7939d14eb15f5cba74e2bdf6f677bbc"
server_key_version: "encryptionkey@0"
I20260620 04:40:08.927911 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:08.942044 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:08.945210 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:08.945418 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:08.953015 22413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36383
I20260620 04:40:08.953043 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36383 every 8 connection(s)
I20260620 04:40:08.953910 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:08.956543 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:08.961943 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: Bootstrap starting.
I20260620 04:40:08.962477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22413
I20260620 04:40:08.962608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:08.962903 392 external_mini_cluster.cc:1428] Setting key 0aa2fd56b7ab12512e6eab5f0abf1806
I20260620 04:40:08.964643 22481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:08.965502 22481 log.cc:824] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:08.967599 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: No bootstrap required, opened a new log
I20260620 04:40:08.971683 22481 raft_consensus.cc:348] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } }
I20260620 04:40:08.971944 22481 raft_consensus.cc:374] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:08.971992 22481 raft_consensus.cc:729] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caac826ddfcf4316b898ff39acb53c61, State: Initialized, Role: FOLLOWER
I20260620 04:40:08.972468 22481 consensus_queue.cc:260] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } }
I20260620 04:40:08.972622 22481 raft_consensus.cc:388] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:08.972688 22481 raft_consensus.cc:482] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:08.972724 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:08.964170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52281 (local address 127.0.98.62:36383)
0620 04:40:08.964649 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:08.964669 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:08.964712 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:40:08.965562 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:08.965585 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:08.965996 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:08.966319 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:08.966704 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:08.968097 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:08.969047 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:08.969721 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:08.969895 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:08.970710 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:08.970716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:08.970740 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:08.971218 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:08.971283 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:08.971312 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:08.971413 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:08.971587 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 04:40:08.971793 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260620 04:40:08.972802 22481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:08.973690 22481 raft_consensus.cc:504] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } }
I20260620 04:40:08.974023 22481 leader_election.cc:302] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caac826ddfcf4316b898ff39acb53c61; no voters:
I20260620 04:40:08.974326 22481 leader_election.cc:288] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:08.974422 22486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:08.974661 22486 raft_consensus.cc:686] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [term 1 LEADER]: Becoming Leader. State: Replica: caac826ddfcf4316b898ff39acb53c61, State: Running, Role: LEADER
I20260620 04:40:08.974994 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } }
I20260620 04:40:08.975476 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:08.975613 22481 sys_catalog.cc:564] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:08.976698 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caac826ddfcf4316b898ff39acb53c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } } }
I20260620 04:40:08.976864 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:08.977134 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caac826ddfcf4316b898ff39acb53c61. Latest consensus state: current_term: 1 leader_uuid: "caac826ddfcf4316b898ff39acb53c61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caac826ddfcf4316b898ff39acb53c61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36383 } } }
I20260620 04:40:08.977223 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61 [sys.catalog]: This master's current role is: LEADER
W20260620 04:40:08.980724 22505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:08.980821 22505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:08.980926 22499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:08.981583 22499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:08.982996 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:08.977814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45731 (local address 127.0.98.62:36383)
0620 04:40:08.977924 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:08.977930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:08.978433 (+ 503us) server_negotiation.cc:407] Connection header received
0620 04:40:08.978514 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:08.978517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:08.978580 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:08.978656 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:08.978873 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:08.979577 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:08.980870 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:08.981173 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:08.981249 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:08.982549 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:08.982552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:08.982555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:08.982567 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:08.982595 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:08.982608 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:08.982691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:08.982824 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 04:40:08.982877 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:40:08.987709 22499 catalog_manager.cc:1347] Generated new cluster ID: 9e376e7e163c45e59280196ed53c46f8
I20260620 04:40:08.987807 22499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:08.998986 22499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:09.000154 22499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:09.005364 22499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caac826ddfcf4316b898ff39acb53c61: Generated new TSK 0
I20260620 04:40:09.006125 22499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:09.013195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22413
2026-06-20T04:40:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:09 dist-test-slave-zmf4 krb5kdc[22512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:09 dist-test-slave-zmf4 krb5kdc[22512](info): set up 2 sockets
Jun 20 04:40:09 dist-test-slave-zmf4 krb5kdc[22512](info): commencing operation
krb5kdc: starting...
W20260620 04:40:10.088438 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:10 dist-test-slave-zmf4 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:10.216009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:10.327955 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:10.328306 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:10.328364 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:10.332347 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:10.332401 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:10.332422 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:10.332438 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:10.332454 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:10.337776 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:10.338768 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:10.339712 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:10.348018 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:10.348018 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:10.348102 22525 server_base.cc:1034] running on GCE node
W20260620 04:40:10.348289 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:10.348917 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:10.349979 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:10.351282 22525 hybrid_clock.cc:648] HybridClock initialized: now 1781930410351256 us; error 38 us; skew 500 ppm
Jun 20 04:40:10 dist-test-slave-zmf4 krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:10.353941 22525 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:10.354774 22525 webserver.cc:460] Webserver started at http://127.0.98.62:33241/ using document root <none> and password file <none>
I20260620 04:40:10.355300 22525 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:10.355386 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:10.355608 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:10.357614 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "b3ee2003d43640b6a8baf57ecf17dea0"
format_stamp: "Formatted at 2026-06-20 04:40:10 on dist-test-slave-zmf4"
server_key: "96c38052429172d2b7036ff3e10dacbb"
server_key_iv: "18e048234e0483ae85046fbdf9a0450f"
server_key_version: "encryptionkey@0"
I20260620 04:40:10.358165 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3ee2003d43640b6a8baf57ecf17dea0"
format_stamp: "Formatted at 2026-06-20 04:40:10 on dist-test-slave-zmf4"
server_key: "96c38052429172d2b7036ff3e10dacbb"
server_key_iv: "18e048234e0483ae85046fbdf9a0450f"
server_key_version: "encryptionkey@0"
I20260620 04:40:10.361706 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:40:10.364022 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:10.365331 22525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:40:10.365475 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "b3ee2003d43640b6a8baf57ecf17dea0"
format_stamp: "Formatted at 2026-06-20 04:40:10 on dist-test-slave-zmf4"
server_key: "96c38052429172d2b7036ff3e10dacbb"
server_key_iv: "18e048234e0483ae85046fbdf9a0450f"
server_key_version: "encryptionkey@0"
I20260620 04:40:10.365588 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:10.370052 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:10.373098 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:10.373303 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:10.380144 22525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33345
I20260620 04:40:10.380167 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33345 every 8 connection(s)
I20260620 04:40:10.381033 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:10.382474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22525
I20260620 04:40:10.382620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:10.382936 392 external_mini_cluster.cc:1428] Setting key bce9aa7868bb58f89d2945d9cb278691
I20260620 04:40:10.384114 22593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:10.387849 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:10.384307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44635 (local address 127.0.98.62:33345)
0620 04:40:10.384749 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:10.384763 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:10.384797 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:40:10.385412 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:10.385426 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:10.385687 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:10.386025 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:10.386764 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:10.386769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:10.386788 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:10.386799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:10.386849 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:10.386867 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:10.387011 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:10.387078 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 04:40:10.387216 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260620 04:40:10.390081 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0: Bootstrap starting.
I20260620 04:40:10.390692 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:10.392372 22593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:10.393285 22593 log.cc:824] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:10.394320 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:10.392721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55077 (local address 127.0.98.62:33345)
0620 04:40:10.392840 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:10.392844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:10.392978 (+ 134us) server_negotiation.cc:407] Connection header received
0620 04:40:10.393051 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:10.393054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:10.393092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:10.393177 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:10.393898 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:10.393900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:10.393902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:40:10.393912 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:10.393936 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:40:10.393946 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:10.394021 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:10.394166 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 04:40:10.394205 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:40:10.395452 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0: No bootstrap required, opened a new log
I20260620 04:40:10.398833 22593 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } }
I20260620 04:40:10.399030 22593 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:40:10.399063 22593 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ee2003d43640b6a8baf57ecf17dea0, State: Initialized, Role: FOLLOWER
I20260620 04:40:10.399488 22593 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } }
I20260620 04:40:10.399612 22593 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:40:10.399703 22593 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:40:10.399816 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:10.400664 22593 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } }
I20260620 04:40:10.400979 22593 leader_election.cc:302] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ee2003d43640b6a8baf57ecf17dea0; no voters:
I20260620 04:40:10.401263 22593 leader_election.cc:288] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:40:10.401355 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:10.401603 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [term 1 LEADER]: Becoming Leader. State: Replica: b3ee2003d43640b6a8baf57ecf17dea0, State: Running, Role: LEADER
I20260620 04:40:10.401924 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } }
I20260620 04:40:10.402462 22593 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:10.403483 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } } }
I20260620 04:40:10.403605 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:10.403961 22612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:10.406956 22612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:10.403924 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ee2003d43640b6a8baf57ecf17dea0. Latest consensus state: current_term: 1 leader_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ee2003d43640b6a8baf57ecf17dea0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33345 } } }
I20260620 04:40:10.407171 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:10.412322 22612 catalog_manager.cc:1347] Generated new cluster ID: 8bf4d28a1c704e2ab44f6608d13eb08b
I20260620 04:40:10.412391 22612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:10.423911 22612 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:10.424276 22612 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:10.433359 22612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ee2003d43640b6a8baf57ecf17dea0: Generated new TSK 0
I20260620 04:40:10.433995 22612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:40:10.437645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22525
2026-06-20T04:40:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1423 ms)
[----------] 18 tests from AuthTokenIssuingTest (25988 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:10 dist-test-slave-zmf4 krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:10 dist-test-slave-zmf4 krb5kdc[22624](info): set up 2 sockets
Jun 20 04:40:10 dist-test-slave-zmf4 krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260620 04:40:11.492540 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:11 dist-test-slave-zmf4 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:11.616271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:11.723800 22637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:11.724052 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:11.724076 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:11.724113 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:11.728018 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:11.728070 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:11.728088 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:11.728106 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:11.728120 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:11.733209 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:11.734107 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:11.735002 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:11.743119 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:11.743129 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:11.743129 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:11.743402 22637 server_base.cc:1034] running on GCE node
I20260620 04:40:11.744029 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:11.745085 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:11.746299 22637 hybrid_clock.cc:648] HybridClock initialized: now 1781930411746274 us; error 38 us; skew 500 ppm
Jun 20 04:40:11 dist-test-slave-zmf4 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:11.749040 22637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:11.749902 22637 webserver.cc:460] Webserver started at http://127.0.98.62:40189/ using document root <none> and password file <none>
I20260620 04:40:11.750399 22637 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:11.750458 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:11.750612 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:11.752590 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "599f898afe1b4e188eb04681f6eea304"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "93658259ddf8a8cb277dc80e220d2ff7"
server_key_iv: "fe18cf2a40bf51153652406d5d0cea20"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.753094 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "599f898afe1b4e188eb04681f6eea304"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "93658259ddf8a8cb277dc80e220d2ff7"
server_key_iv: "fe18cf2a40bf51153652406d5d0cea20"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.756695 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:40:11.759106 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:11.760406 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:40:11.760538 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "599f898afe1b4e188eb04681f6eea304"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "93658259ddf8a8cb277dc80e220d2ff7"
server_key_iv: "fe18cf2a40bf51153652406d5d0cea20"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.760650 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:11.774214 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:11.777307 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:11.777580 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:11.786653 22637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41177
I20260620 04:40:11.786886 22704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41177 every 8 connection(s)
I20260620 04:40:11.787734 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:11.790506 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:11.792728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22637
I20260620 04:40:11.792853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:11.793169 392 external_mini_cluster.cc:1428] Setting key b94fa873f7d282e10d57e224082705dd
I20260620 04:40:11.794214 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:40:11 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:11.809827 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.796476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.62:41177)
0620 04:40:11.796909 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:11.796927 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:11.797210 (+ 283us) server_negotiation.cc:407] Connection header received
0620 04:40:11.798213 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:11.798217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:11.798417 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:11.798617 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:11.799441 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:11.800361 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:11.801369 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:11.801764 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:11.802023 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:11.804702 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:11.804712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:11.804728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:11.804759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:11.806645 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:11.807263 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:11.807268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:11.807269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:11.807375 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:11.807777 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:11.807780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:11.807781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:11.808157 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:11.808337 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:11.808809 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 04:40:11.809064 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260620 04:40:11.809830 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.796477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41177 (local address 127.0.0.1:45246)
0620 04:40:11.797059 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.797095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:11.797913 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:11.798611 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:11.798616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:11.799065 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:11.799288 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:11.799300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:11.800477 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:11.800480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:11.801250 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:11.801260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:11.801915 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:11.801919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:11.802140 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:11.802804 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:11.802828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:11.804537 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:11.806762 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:11.806774 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:11.806787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:11.807121 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:11.807484 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:11.807490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:11.807495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:11.807655 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:11.808331 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:11.808341 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:11.808526 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 04:40:11.808740 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 04:40:11.809170 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":132,"spinlock_wait_cycles":42752,"thread_start_us":162,"threads_started":1}
I20260620 04:40:11.812408 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:11.813158 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.812905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38654)
0620 04:40:11.813025 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.813100 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:40:11.813342 22654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
I20260620 04:40:11.814174 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.813907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38660)
0620 04:40:11.814013 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.814062 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:40:11.814311 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41803: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:40:11.821245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41803
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:40:11.863585 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } attempt: 1
I20260620 04:40:11.864450 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.864146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38674)
0620 04:40:11.864306 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.864378 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 04:40:11.865152 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.864950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38690)
0620 04:40:11.865058 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.865105 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 04:40:11.865279 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41803: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
W20260620 04:40:11.931299 22712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:11.931578 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:11.931602 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:11.931687 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:40:11.933466 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } attempt: 2
I20260620 04:40:11.934295 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.934014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38704)
0620 04:40:11.934159 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.934225 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 04:40:11.935067 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:11.934842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38714)
0620 04:40:11.934973 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:11.935019 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 04:40:11.935336 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41803: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41803: connect: Connection refused (error 111)
W20260620 04:40:11.936457 22712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:11.936517 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:11.936540 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:11.936559 22712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:11.936663 22712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:11.942106 22712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:11.943218 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:11.944164 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:11.952584 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:11.952626 22712 server_base.cc:1034] running on GCE node
W20260620 04:40:11.952592 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:11.952612 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:11.953413 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:11.954387 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:11.955581 22712 hybrid_clock.cc:648] HybridClock initialized: now 1781930411955554 us; error 40 us; skew 500 ppm
Jun 20 04:40:11 dist-test-slave-zmf4 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:11.958534 22712 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:40:11.959360 22712 webserver.cc:460] Webserver started at http://127.0.98.61:39337/ using document root <none> and password file <none>
I20260620 04:40:11.959887 22712 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:11.959946 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:11.960173 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:11.962177 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data/instance:
uuid: "09d84c7a2eff4970b857c0c9332b9540"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "6269ea8f9d3bbb5999de36e6156d9ae9"
server_key_iv: "b4083ff41db55c84a97e8ee91d5e48a4"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.962739 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal/instance:
uuid: "09d84c7a2eff4970b857c0c9332b9540"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "6269ea8f9d3bbb5999de36e6156d9ae9"
server_key_iv: "b4083ff41db55c84a97e8ee91d5e48a4"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.966516 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:40:11.968956 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:11.970170 22712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:40:11.970320 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "09d84c7a2eff4970b857c0c9332b9540"
format_stamp: "Formatted at 2026-06-20 04:40:11 on dist-test-slave-zmf4"
server_key: "6269ea8f9d3bbb5999de36e6156d9ae9"
server_key_iv: "b4083ff41db55c84a97e8ee91d5e48a4"
server_key_version: "encryptionkey@0"
I20260620 04:40:11.970435 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:12.004321 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:12.007866 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:12.008086 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:12.015367 22712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41803
I20260620 04:40:12.015653 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41803 every 8 connection(s)
I20260620 04:40:12.016417 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:40:12.018610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22712
I20260620 04:40:12.018755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-1/wal/instance
I20260620 04:40:12.019047 392 external_mini_cluster.cc:1428] Setting key 4843c0a5b7119173b3f41ccc3f47b0c3
I20260620 04:40:12.019460 22780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:12.023658 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 04:40:12.026530 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:12.038874 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.026937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38728)
0620 04:40:12.027074 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.027089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.027164 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.028076 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.028079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.028296 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.028425 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.028432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.030141 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.030144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.030929 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.030938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.031550 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.031555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.031644 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.032262 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.032285 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.033806 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.036660 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.036664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.036666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.037091 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.037494 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.037504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.037598 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.038320 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.038324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.038411 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.038541 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.038681 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 04:40:12.039362 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.025766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.62:41177)
0620 04:40:12.025939 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.025947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.026673 (+ 726us) server_negotiation.cc:407] Connection header received
0620 04:40:12.027825 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.027828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.027877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.027973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.028934 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.029634 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.030894 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.031306 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.031432 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.034621 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.034628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.034632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.034664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.037180 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037714 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.037717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.037719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.037810 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.038192 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.038196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.038198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.038426 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.038506 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.038920 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.039033 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:40:12.040020 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.026071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41177 (local address 127.0.0.1:45262)
0620 04:40:12.026529 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.026577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.027626 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.028046 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.028053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.028514 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.028803 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.028813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.029749 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.029754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.030769 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.030779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.031465 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.031471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.031641 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.032452 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.032475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.034490 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.037294 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.037309 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.037610 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.037945 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.037949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.038040 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.038545 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.038552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.038681 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.038818 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.039193 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":43,"thread_start_us":148,"threads_started":1}
I20260620 04:40:12.040019 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.027154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.98.61:41803)
0620 04:40:12.027457 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.027465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.027550 (+ 85us) server_negotiation.cc:407] Connection header received
0620 04:40:12.027681 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.027695 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.027916 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.028094 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.028662 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.030007 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.031037 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.031421 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.031882 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.033972 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.033981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.033999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.034033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.036535 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037227 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.037235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.037243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.037379 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.037714 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.037717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.037719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.038223 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.038358 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.038678 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.039039 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":204,"thread_start_us":127,"threads_started":1}
I20260620 04:40:12.041312 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:12.041858 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:12.042454 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.042243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42774)
0620 04:40:12.042345 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.042393 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:40:12.043186 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.042877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42780)
0620 04:40:12.043086 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.043140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 04:40:12.043382 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:40:12.047369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46679
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:40:12.050022 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.041799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38738)
0620 04:40:12.041908 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.041922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.041994 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.042274 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.042276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.042433 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.042529 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.042539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.043513 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.043516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.044050 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.044058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.044615 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.044618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.044733 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.045129 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.045138 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.046498 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.048517 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.048523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.048526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.048725 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.049013 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.049018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.049020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.049105 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.049486 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.049489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.049573 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.049691 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.049840 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 04:40:12.050032 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.041972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.98.61:41803)
0620 04:40:12.042062 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.042072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.042082 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:40:12.042132 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.042137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.042184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.042260 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.042642 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.043411 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.044163 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.044511 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.044620 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.046669 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.046672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.046674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.046694 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.048407 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.048831 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.048838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.048840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.048915 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.049206 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.049209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.049211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.049383 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.049498 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.049747 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.049899 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":33}
I20260620 04:40:12.050952 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:12.051712 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.051379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42782)
0620 04:40:12.051523 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.051621 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:40:12.051949 22729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
I20260620 04:40:12.052966 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.052664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42786)
0620 04:40:12.052808 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.052871 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 04:40:12.053174 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
I20260620 04:40:12.074541 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 1
I20260620 04:40:12.075348 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.075031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42794)
0620 04:40:12.075190 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.075262 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 04:40:12.076162 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.075943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42796)
0620 04:40:12.076050 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.076103 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:40:12.076320 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
I20260620 04:40:12.102459 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 1
I20260620 04:40:12.103433 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.103017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42812)
0620 04:40:12.103178 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.103324 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 04:40:12.104581 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.104342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42822)
0620 04:40:12.104458 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.104515 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 04:40:12.104754 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
I20260620 04:40:12.151515 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 2
I20260620 04:40:12.152438 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.152045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42828)
0620 04:40:12.152235 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.152317 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 04:40:12.153191 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.152960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42838)
0620 04:40:12.153074 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.153127 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 04:40:12.153379 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
W20260620 04:40:12.163297 22786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:12.163715 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:12.163864 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:12.163995 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:12.168653 22786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:12.168710 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:12.168730 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:12.168747 22786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:12.168762 22786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:12.172950 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 2
I20260620 04:40:12.173926 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.173559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42852)
0620 04:40:12.173760 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.173829 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 04:40:12.174571 22786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41177,127.0.98.61:41803,127.0.98.60:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:12.174747 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.174486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42862)
0620 04:40:12.174596 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.174648 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:40:12.174899 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46679: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46679: connect: Connection refused (error 111)
I20260620 04:40:12.175704 22786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:12.176790 22786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:12.185391 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:12.185420 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:12.185420 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:12.185854 22786 server_base.cc:1034] running on GCE node
I20260620 04:40:12.186410 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:12.187448 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:12.188606 22786 hybrid_clock.cc:648] HybridClock initialized: now 1781930412188569 us; error 54 us; skew 500 ppm
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:12.191269 22786 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:40:12.192101 22786 webserver.cc:460] Webserver started at http://127.0.98.60:34573/ using document root <none> and password file <none>
I20260620 04:40:12.192657 22786 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:12.192718 22786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:12.192873 22786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:12.194792 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data/instance:
uuid: "9850f41f44824237ae25f546aac7b8fe"
format_stamp: "Formatted at 2026-06-20 04:40:12 on dist-test-slave-zmf4"
server_key: "04d5088b0286e16de97d125061edcf5a"
server_key_iv: "f284df9ac766c2eaf913e28d8d7b6b9f"
server_key_version: "encryptionkey@0"
I20260620 04:40:12.195784 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal/instance:
uuid: "9850f41f44824237ae25f546aac7b8fe"
format_stamp: "Formatted at 2026-06-20 04:40:12 on dist-test-slave-zmf4"
server_key: "04d5088b0286e16de97d125061edcf5a"
server_key_iv: "f284df9ac766c2eaf913e28d8d7b6b9f"
server_key_version: "encryptionkey@0"
I20260620 04:40:12.199338 22786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:40:12.201727 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:12.203008 22786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:40:12.203159 22786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "9850f41f44824237ae25f546aac7b8fe"
format_stamp: "Formatted at 2026-06-20 04:40:12 on dist-test-slave-zmf4"
server_key: "04d5088b0286e16de97d125061edcf5a"
server_key_iv: "f284df9ac766c2eaf913e28d8d7b6b9f"
server_key_version: "encryptionkey@0"
I20260620 04:40:12.203274 22786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:12.220322 22786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:12.223595 22786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:12.224098 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:12.231616 22786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46679
I20260620 04:40:12.232050 22854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46679 every 8 connection(s)
I20260620 04:40:12.232877 22786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:40:12.234141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 22786
I20260620 04:40:12.234288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781930128116734-392-0/minicluster-data/master-2/wal/instance
I20260620 04:40:12.234786 392 external_mini_cluster.cc:1428] Setting key 2eff22a128accb47c357387a4bc7e570
I20260620 04:40:12.237321 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:12.242292 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:12.251580 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.237668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.60:46679)
0620 04:40:12.238155 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.238174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.238221 (+ 47us) server_negotiation.cc:407] Connection header received
0620 04:40:12.238872 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.238887 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.239379 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.239662 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.240784 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.241792 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.243118 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.243587 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.243708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.246642 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.246647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.246657 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.246683 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.248736 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.249190 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.249195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.249199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.249310 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.249601 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.249604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.249605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.250103 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.250309 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.250582 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.250841 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":182,"threads_started":1}
I20260620 04:40:12.252533 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 3
I20260620 04:40:12.253661 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.244287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41177 (local address 127.0.0.1:45270)
0620 04:40:12.244695 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.244739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.245044 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.245376 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.245381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.245725 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.245849 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.245855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.246648 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.246652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.247420 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.247435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.247907 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.247911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.248024 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.248502 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.248514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.249957 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.251951 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.251957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.251962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.252257 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.252526 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.252529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.252531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.252625 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.253070 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.253074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.253160 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.253261 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.253467 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260620 04:40:12.253677 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.244313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.0.98.62:41177)
0620 04:40:12.244459 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.244464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.244873 (+ 409us) server_negotiation.cc:407] Connection header received
0620 04:40:12.245181 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.245184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.245238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.245331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.245979 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.246528 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.247548 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.247793 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.247906 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.250104 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.250108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.250109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.250132 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.251833 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.252356 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.252359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.252360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.252438 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.252739 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.252742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.252744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.252952 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.253017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.253406 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.253529 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:40:12.255424 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:40:12.264708 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.253517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42878)
0620 04:40:12.253754 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.253770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.253845 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.254299 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.254302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.254487 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.254580 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.254585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.255557 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.255560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.256362 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.256374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.256955 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.256959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.257043 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.257592 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.257606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.259059 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.261776 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.261779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.261781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.262172 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.263369 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.263374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.263376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.263482 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.263885 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.263888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.263989 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.264192 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.264512 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 04:40:12.265527 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.253969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.0.98.60:46679)
0620 04:40:12.254074 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.254078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.254091 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:40:12.254125 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.254127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.254168 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.254665 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.254746 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.255439 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.256487 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.256837 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.256942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.259198 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.259202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.259206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.259233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.261634 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.262771 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.262787 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.262789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.262974 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.263584 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.263586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.263587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.263750 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.265159 (+ 1409us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.265230 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.265355 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":29}
I20260620 04:40:12.265952 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.254187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.98.62:41177)
0620 04:40:12.254857 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.254863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.254880 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:40:12.254910 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.254912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.254950 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.255046 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.255817 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.256638 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.257733 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.258052 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.258144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.261306 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.261313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.261316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.261340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.263330 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.263828 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.263832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.263836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.263914 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.264276 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.264280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.264281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.264792 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.265039 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.265710 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.265821 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611}
I20260620 04:40:12.266427 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } attempt: 3
I20260620 04:40:12.267253 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.256069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.61:41803)
0620 04:40:12.256247 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.256251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.256397 (+ 146us) server_negotiation.cc:407] Connection header received
0620 04:40:12.256513 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.256516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.256561 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.256635 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.257118 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.257590 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.258364 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.259255 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.259391 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.262162 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.262166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.262168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.262195 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.264036 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.265179 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.265183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.265185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.265282 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.266048 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.266054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.266057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.266562 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.266660 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.266963 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.267087 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:40:12.267721 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.255903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41803 (local address 127.0.0.1:38750)
0620 04:40:12.256313 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.256327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.256409 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.256660 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.256663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.256846 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.256956 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.256962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.257705 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.257711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.258239 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.258252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.259543 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.259548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.259669 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.260128 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.260146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.262014 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.264650 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.264653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.264655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.265039 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.265364 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.265366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.265368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.265451 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.266653 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.266656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.266767 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.267479 (+ 712us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.267619 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260620 04:40:12.268350 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:12.269352 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: Bootstrap starting.
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:40:12.272356 22705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:12.273485 22705 log.cc:824] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:40:12.277070 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.266860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42892)
0620 04:40:12.267037 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.267053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.267135 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.267389 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.267392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.267549 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.267637 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.267643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.268650 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.268656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.269543 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.269556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.270345 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.270349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.270431 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.270922 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.270937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.272571 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.275375 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.275381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.275386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.275627 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.275942 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.275946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.275947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.276039 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.276580 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.276583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.276679 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.276797 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.276921 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 04:40:12.277518 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.266893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.0.98.60:46679)
0620 04:40:12.266998 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.267002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.267121 (+ 119us) server_negotiation.cc:407] Connection header received
0620 04:40:12.267211 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.267216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.267264 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.267737 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.267832 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.268537 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.269660 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.270208 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.270333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.272730 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.272734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.272736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.272790 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.275122 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.275735 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.275739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.275740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.275839 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.276155 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.276161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.276165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.276431 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.276536 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.277280 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.277389 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:40:12.277962 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: No bootstrap required, opened a new log
I20260620 04:40:12.278468 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.268824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46679 (local address 127.0.0.1:42898)
0620 04:40:12.269428 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:12.269442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:12.269515 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:12.269746 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:12.269749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:12.269926 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:12.269988 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.269993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.271265 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.271267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.271876 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:12.271885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.272504 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.272508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:12.272584 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.273050 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:12.273064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:12.274726 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:12.277050 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.277053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.277055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.277290 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.277553 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:12.277556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:12.277558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:12.277647 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:12.278024 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:12.278029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:12.278103 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 04:40:12.278187 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:40:12.278323 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544}
I20260620 04:40:12.279031 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.268897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.0.98.60:46679)
0620 04:40:12.269206 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.269215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.269584 (+ 369us) server_negotiation.cc:407] Connection header received
0620 04:40:12.269616 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.269620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.269664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.269737 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.270102 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.271151 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.271994 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.272347 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.272461 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.274843 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.274846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.274848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.274875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.276934 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.277388 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.277391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.277392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.277471 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.277724 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.277727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.277729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.277900 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.277991 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.278727 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.278838 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260620 04:40:12.280529 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: Bootstrap starting.
Jun 20 04:40:12 dist-test-slave-zmf4 krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:40:12.282023 22705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.282261 22705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599f898afe1b4e188eb04681f6eea304, State: Initialized, Role: FOLLOWER
I20260620 04:40:12.282575 22705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.283257 22780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:12.283952 22705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:12.284094 22780 log.cc:824] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:12.285737 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.274252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.0.98.61:41803)
0620 04:40:12.274378 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.274382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.274396 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:40:12.274572 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.274577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.274624 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.274713 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.275719 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.276604 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.278355 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.278697 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.278769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.281906 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.281910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.281912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.281934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.283737 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.284308 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.284312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.284313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.284406 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.284743 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.284747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.284748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.284971 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.285108 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.285474 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.285589 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:40:12.286558 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: No bootstrap required, opened a new log
I20260620 04:40:12.287376 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: Bootstrap starting.
I20260620 04:40:12.289981 22855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:12.290627 22780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.290808 22855 log.cc:824] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:12.290838 22780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d84c7a2eff4970b857c0c9332b9540, State: Initialized, Role: FOLLOWER
I20260620 04:40:12.291159 22780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.291613 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.291750 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.292501 22780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:12.288846 22637 master_runner.cc:186] Error getting master registration for 127.0.98.61:41803: OK, instance_id { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" instance_seqno: 1781930412013461 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:40:12.292898 22637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:40:12.292986 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: No bootstrap required, opened a new log
I20260620 04:40:12.293525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:12.294775 22679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "599f898afe1b4e188eb04681f6eea304"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45286
I20260620 04:40:12.294962 22679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:40:12.295048 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 0 FOLLOWER]: Advancing to term 1
W20260620 04:40:12.296579 22874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:12.296651 22874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:12.297305 22679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.297549 22855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.297787 22855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9850f41f44824237ae25f546aac7b8fe, State: Initialized, Role: FOLLOWER
I20260620 04:40:12.297734 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.297977 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.298110 22855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.298298 22679 leader_election.cc:288] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [CANDIDATE]: Term 1 election: Requested vote from peers 09d84c7a2eff4970b857c0c9332b9540 (127.0.98.61:41803), 9850f41f44824237ae25f546aac7b8fe (127.0.98.60:46679)
I20260620 04:40:12.298764 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "599f898afe1b4e188eb04681f6eea304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09d84c7a2eff4970b857c0c9332b9540"
I20260620 04:40:12.298995 22753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:12.300351 22753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599f898afe1b4e188eb04681f6eea304 in term 1.
I20260620 04:40:12.300776 22654 leader_election.cc:302] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d84c7a2eff4970b857c0c9332b9540, 599f898afe1b4e188eb04681f6eea304; no voters:
I20260620 04:40:12.301040 22862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:12.301242 22862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [term 1 LEADER]: Becoming Leader. State: Replica: 599f898afe1b4e188eb04681f6eea304, State: Running, Role: LEADER
I20260620 04:40:12.301532 22862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } }
I20260620 04:40:12.302021 22855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:12.303287 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599f898afe1b4e188eb04681f6eea304. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.303390 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:12.303954 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "599f898afe1b4e188eb04681f6eea304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9850f41f44824237ae25f546aac7b8fe"
I20260620 04:40:12.304982 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:12.306303 22829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599f898afe1b4e188eb04681f6eea304 in term 1.
W20260620 04:40:12.307410 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:12.307495 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:12.307876 22903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:12.308899 22903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:12.310595 22712 master_runner.cc:263] No leader master found from master 09d84c7a2eff4970b857c0c9332b9540
I20260620 04:40:12.310657 22712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:40:12.315506 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.301869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.0.98.61:41803)
0620 04:40:12.301975 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.301979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.302933 (+ 954us) server_negotiation.cc:407] Connection header received
0620 04:40:12.303115 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.303121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.303178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.303227 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.304325 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.305063 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.306464 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.306781 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.306859 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.309682 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.309685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.309688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.309712 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.312401 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.314133 (+ 1732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.314137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.314139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.314225 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.314550 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.314554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.314555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.314772 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.314875 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.315225 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.315350 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:40:12.315668 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.315797 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.317335 22829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [term 1 FOLLOWER]: Refusing update from remote peer 599f898afe1b4e188eb04681f6eea304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:40:12.318053 22753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [term 1 FOLLOWER]: Refusing update from remote peer 599f898afe1b4e188eb04681f6eea304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:40:12.320045 22862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:40:12.320564 22862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:40:12.324597 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599f898afe1b4e188eb04681f6eea304. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.324707 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.328045 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599f898afe1b4e188eb04681f6eea304. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.328176 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.328835 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.307869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.98.62:41177)
0620 04:40:12.307973 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.307978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.316627 (+ 8649us) server_negotiation.cc:407] Connection header received
0620 04:40:12.316669 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.316671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.316717 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.316819 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.318902 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.319794 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.322986 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.323295 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.323400 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.325148 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.325154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.325159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.325187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.326842 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.327323 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.327327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.327328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.327409 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.327680 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.327683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.327685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.327882 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.327996 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.328563 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.328677 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:40:12.335888 22887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:12.335980 22887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:12.340216 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.340386 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:12.341799 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.341943 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.342525 22912 mvcc.cc:205] Tried to move back new op lower bound from 7298786968843259904 to 7298786968790417408. Current Snapshot: MvccSnapshot[applied={T|T < 7298786968843259904}]
I20260620 04:40:12.343017 22913 mvcc.cc:205] Tried to move back new op lower bound from 7298786968843259904 to 7298786968790417408. Current Snapshot: MvccSnapshot[applied={T|T < 7298786968843259904}]
I20260620 04:40:12.343701 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.343942 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:12.345302 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "599f898afe1b4e188eb04681f6eea304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599f898afe1b4e188eb04681f6eea304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } peers { permanent_uuid: "09d84c7a2eff4970b857c0c9332b9540" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41803 } } peers { permanent_uuid: "9850f41f44824237ae25f546aac7b8fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46679 } } }
I20260620 04:40:12.345427 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:12.345626 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:12.302458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.0.98.60:46679)
0620 04:40:12.302572 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.302576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.313045 (+ 10469us) server_negotiation.cc:407] Connection header received
0620 04:40:12.313212 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.313215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.313273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.313326 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.323558 (+ 10232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.324373 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.325481 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:12.325801 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:12.325875 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:12.327600 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:12.327605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:12.327607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:12.327634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:12.329448 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.336641 (+ 7193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.336646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.336647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.336766 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:12.344515 (+ 7749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:12.344518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:12.344520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:12.344761 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:12.344907 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:12.345317 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 04:40:12.345458 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:40:12.347277 22903 catalog_manager.cc:1347] Generated new cluster ID: c92db560ae8441c5a901d025101e6bb6
I20260620 04:40:12.347347 22903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:12.363703 22903 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:12.365110 22903 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:12.381052 22903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599f898afe1b4e188eb04681f6eea304: Generated new TSK 0
I20260620 04:40:12.382026 22903 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:40:12.457729 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.444714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.0.98.61:41803)
0620 04:40:12.444859 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.444864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.456537 (+ 11673us) server_negotiation.cc:407] Connection header received
0620 04:40:12.456807 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.456812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.456873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.456928 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.457589 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38784: BlockingRecv error: recv got EOF from 127.0.0.1:38784 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 04:40:12.458379 22860 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.445502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.98.60:46679)
0620 04:40:12.445638 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.445643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.456552 (+ 10909us) server_negotiation.cc:407] Connection header received
0620 04:40:12.456818 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.456822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.456874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.456928 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.458262 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42912: BlockingRecv error: recv got EOF from 127.0.0.1:42912 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:40:12.459002 22708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.448537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.98.62:41177)
0620 04:40:12.448658 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.448662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.457470 (+ 8808us) server_negotiation.cc:407] Connection header received
0620 04:40:12.457651 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.457654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.457704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.457757 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.458896 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45308: BlockingRecv error: recv got EOF from 127.0.0.1:45308 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 04:40:12.480640 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.461426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.98.61:41803)
0620 04:40:12.461531 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.461535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.473955 (+ 12420us) server_negotiation.cc:407] Connection header received
0620 04:40:12.474685 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.474690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.474827 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.474914 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.480514 (+ 5600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 04:40:12.505860 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.489094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.98.61:41803)
0620 04:40:12.489240 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.489244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.504588 (+ 15344us) server_negotiation.cc:407] Connection header received
0620 04:40:12.505018 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.505023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.505088 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.505151 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.505735 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38790: BlockingRecv error: recv got EOF from 127.0.0.1:38790 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 04:40:12.533757 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.516659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.0.98.61:41803)
0620 04:40:12.516923 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.516928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.532653 (+ 15725us) server_negotiation.cc:407] Connection header received
0620 04:40:12.532915 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.532919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.532981 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.533037 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.533636 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38806: BlockingRecv error: recv got EOF from 127.0.0.1:38806 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 04:40:12.550051 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.540575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.98.61:41803)
0620 04:40:12.540727 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.540744 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.548957 (+ 8213us) server_negotiation.cc:407] Connection header received
0620 04:40:12.549199 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.549204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.549265 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.549329 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.549929 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38820: BlockingRecv error: recv got EOF from 127.0.0.1:38820 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 04:40:12.573969 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.572113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.0.98.61:41803)
0620 04:40:12.572288 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.572295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.572750 (+ 455us) server_negotiation.cc:407] Connection header received
0620 04:40:12.572997 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.573003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.573076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.573137 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.573807 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38830: BlockingRecv error: recv got EOF from 127.0.0.1:38830 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 04:40:12.598371 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.596677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.0.98.61:41803)
0620 04:40:12.596822 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.596827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.597418 (+ 591us) server_negotiation.cc:407] Connection header received
0620 04:40:12.597651 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.597655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.597714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.597811 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.598246 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38838: BlockingRecv error: recv got EOF from 127.0.0.1:38838 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 04:40:12.634050 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.632806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.0.98.61:41803)
0620 04:40:12.632951 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.632955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.633139 (+ 184us) server_negotiation.cc:407] Connection header received
0620 04:40:12.633314 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.633320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.633374 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.633473 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.633907 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38848: BlockingRecv error: recv got EOF from 127.0.0.1:38848 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:40:12.701399 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.700211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.98.61:41803)
0620 04:40:12.700356 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.700361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.700499 (+ 138us) server_negotiation.cc:407] Connection header received
0620 04:40:12.700677 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.700681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.700732 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.700826 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.701296 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38858: BlockingRecv error: recv got EOF from 127.0.0.1:38858 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:40:12.832964 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:12.831692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.61:41803)
0620 04:40:12.831868 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:12.831873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:12.832001 (+ 128us) server_negotiation.cc:407] Connection header received
0620 04:40:12.832224 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:12.832228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:12.832285 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:12.832381 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:12.832854 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38872: BlockingRecv error: recv got EOF from 127.0.0.1:38872 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 04:40:13.092460 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:13.091239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.98.61:41803)
0620 04:40:13.091408 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.091412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.091428 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:40:13.091576 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.091585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.091632 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.091725 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:13.092327 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38880: BlockingRecv error: recv got EOF from 127.0.0.1:38880 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:40:13.308947 22909 catalog_manager.cc:1259] Loaded cluster ID: c92db560ae8441c5a901d025101e6bb6
I20260620 04:40:13.309036 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: loading cluster ID for follower catalog manager: success
I20260620 04:40:13.310734 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: acquiring CA information for follower catalog manager: success
I20260620 04:40:13.311928 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9850f41f44824237ae25f546aac7b8fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:40:13.337529 22887 catalog_manager.cc:1259] Loaded cluster ID: c92db560ae8441c5a901d025101e6bb6
I20260620 04:40:13.337611 22887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: loading cluster ID for follower catalog manager: success
I20260620 04:40:13.339358 22887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: acquiring CA information for follower catalog manager: success
I20260620 04:40:13.340560 22887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09d84c7a2eff4970b857c0c9332b9540: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:40:13.610667 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:13.606562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.61:41803)
0620 04:40:13.606844 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.606848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.606866 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:40:13.607045 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.607049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.607175 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.607291 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:13.607582 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.608165 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.609087 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.609338 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.609429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:13.609779 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:13.610220 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:13.610323 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:13.610489 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 04:40:13.610542 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260620 04:40:13.617723 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:13.613795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.98.60:46679)
0620 04:40:13.614019 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.614022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.614034 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:40:13.614104 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.614107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.614207 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.614317 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:13.614677 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.615315 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.616282 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.616520 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.616609 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:13.616947 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:13.617341 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:13.617436 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:13.617562 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 04:40:13.617609 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260620 04:40:13.625240 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:13.620881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.0.98.61:41803)
0620 04:40:13.621039 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.621043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.621411 (+ 368us) server_negotiation.cc:407] Connection header received
0620 04:40:13.621596 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.621601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.621649 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.621749 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:13.622257 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.623004 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.623918 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.624179 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.624319 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:13.624794 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:13.624872 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:13.624957 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:13.625045 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 04:40:13.625096 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 04:40:13.625237 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:13.621319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.60:46679)
0620 04:40:13.621436 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.621439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.621585 (+ 146us) server_negotiation.cc:407] Connection header received
0620 04:40:13.621762 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.621765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.621811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.621882 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:13.622155 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.622922 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.624026 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.624380 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.624470 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:13.624731 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:13.624802 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:13.624879 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:13.625037 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 04:40:13.625095 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:40:13.625633 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:13.620835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.62:41177)
0620 04:40:13.621039 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:13.621043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:13.621143 (+ 100us) server_negotiation.cc:407] Connection header received
0620 04:40:13.621359 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:13.621363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:13.621522 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:13.621637 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:13.622001 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.622643 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.623830 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:13.624189 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:13.624314 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:13.624628 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:13.624978 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:13.625056 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:13.625435 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 04:40:13.625494 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260620 04:40:13.628350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22637
I20260620 04:40:13.634521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22712
I20260620 04:40:13.640703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 22786
2026-06-20T04:40:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3205 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:40:13 dist-test-slave-zmf4 krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:40:13 dist-test-slave-zmf4 krb5kdc[23004](info): set up 2 sockets
Jun 20 04:40:13 dist-test-slave-zmf4 krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260620 04:40:14.713105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/test-user.keytab.
Jun 20 04:40:14 dist-test-slave-zmf4 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:40:14.841481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:40:14.955396 23017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:14.955693 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:14.955718 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:14.955760 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:14.959964 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:14.960024 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:14.960043 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:14.960060 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:14.960075 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:14.965329 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:14.966393 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:14.967422 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:14.976028 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:14.976049 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:14.976099 23017 server_base.cc:1034] running on GCE node
W20260620 04:40:14.976045 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:14.976934 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:14.978019 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:14.979200 23017 hybrid_clock.cc:648] HybridClock initialized: now 1781930414979181 us; error 27 us; skew 500 ppm
Jun 20 04:40:14 dist-test-slave-zmf4 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:14.986016 23017 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:40:14.986979 23017 webserver.cc:460] Webserver started at http://127.0.98.62:36585/ using document root <none> and password file <none>
I20260620 04:40:14.987542 23017 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:14.987602 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:14.987833 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:14.990090 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data/instance:
uuid: "40cbda1df9b94a02bdd89a3ebe4ef848"
format_stamp: "Formatted at 2026-06-20 04:40:14 on dist-test-slave-zmf4"
server_key: "1b1c7a8bfa4270df961258a991c62285"
server_key_iv: "b620c179e4d2ddbe1023a038e9327e3c"
server_key_version: "encryptionkey@0"
I20260620 04:40:14.990672 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance:
uuid: "40cbda1df9b94a02bdd89a3ebe4ef848"
format_stamp: "Formatted at 2026-06-20 04:40:14 on dist-test-slave-zmf4"
server_key: "1b1c7a8bfa4270df961258a991c62285"
server_key_iv: "b620c179e4d2ddbe1023a038e9327e3c"
server_key_version: "encryptionkey@0"
I20260620 04:40:14.995575 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 04:40:14.998644 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:15.000291 23017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:40:15.000461 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal
uuid: "40cbda1df9b94a02bdd89a3ebe4ef848"
format_stamp: "Formatted at 2026-06-20 04:40:14 on dist-test-slave-zmf4"
server_key: "1b1c7a8bfa4270df961258a991c62285"
server_key_iv: "b620c179e4d2ddbe1023a038e9327e3c"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.000597 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:15.027055 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:15.030359 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:15.030611 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:15.038102 23017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39555
I20260620 04:40:15.038368 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39555 every 8 connection(s)
I20260620 04:40:15.039178 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:40:15.042034 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:15.045761 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.048470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 23017
I20260620 04:40:15.048595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-0/wal/instance
I20260620 04:40:15.048902 392 external_mini_cluster.cc:1428] Setting key 313650a1d0685af5bc387283bbec08af
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 04:40:15.061298 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.048034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.0.98.62:39555)
0620 04:40:15.048786 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.048799 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.048864 (+ 65us) server_negotiation.cc:407] Connection header received
0620 04:40:15.049889 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.049893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.050068 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.050255 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.051074 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.051926 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.052958 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.053318 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.053504 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.056320 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.056325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.056336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.056356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.058206 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.058723 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.058731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.058733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.058847 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.059224 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.059228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.059229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.059616 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.059763 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.060212 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.060465 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":636,"spinlock_wait_cycles":32640,"thread_start_us":346,"threads_started":1}
I20260620 04:40:15.061306 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.048034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39555 (local address 127.0.0.1:36146)
0620 04:40:15.048708 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.048758 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.049574 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.050255 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.050260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.050700 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.050928 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.050941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.052067 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.052071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.052839 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.052848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.053490 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.053493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.053605 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.054248 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.054264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.056091 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.058352 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.058360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.058371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.058599 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.058952 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.058955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.058957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.059123 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.059745 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.059752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.059896 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.060075 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.060570 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":101,"thread_start_us":354,"threads_started":1}
I20260620 04:40:15.063997 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.065146 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.064601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47598)
0620 04:40:15.064866 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.065020 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 04:40:15.065500 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
I20260620 04:40:15.066910 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.066473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47610)
0620 04:40:15.066749 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.066825 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260620 04:40:15.067184 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:40:15.080380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39523
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:40:15.116479 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } attempt: 1
I20260620 04:40:15.117605 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.117284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47620)
0620 04:40:15.117446 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.117519 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 04:40:15.118341 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.118117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47622)
0620 04:40:15.118228 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.118285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 04:40:15.118479 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
I20260620 04:40:15.186664 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } attempt: 2
I20260620 04:40:15.187584 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.187251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47634)
0620 04:40:15.187452 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.187511 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 04:40:15.188391 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.188152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47640)
0620 04:40:15.188287 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.188335 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 04:40:15.188512 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39523: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39523: connect: Connection refused (error 111)
W20260620 04:40:15.197029 23092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:15.197337 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:15.197500 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:15.197628 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:15.201663 23092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:15.201738 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:15.201774 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:15.201829 23092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:15.201864 23092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:15.207551 23092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:15.208758 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:15.209769 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:15.218464 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:15.218506 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:15.218745 23092 server_base.cc:1034] running on GCE node
W20260620 04:40:15.218506 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:15.219295 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:15.220417 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:15.221593 23092 hybrid_clock.cc:648] HybridClock initialized: now 1781930415221575 us; error 31 us; skew 500 ppm
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:15.224637 23092 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:40:15.225431 23092 webserver.cc:460] Webserver started at http://127.0.98.61:39539/ using document root <none> and password file <none>
I20260620 04:40:15.226199 23092 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:15.226287 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:15.226495 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:15.228695 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data/instance:
uuid: "5e2e01d0da7c495d86faf7cad7c99461"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "29614ddd8c1913f303f9f139761cdcfb"
server_key_iv: "cff144bda93391b919e691a07c83c37a"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.229260 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal/instance:
uuid: "5e2e01d0da7c495d86faf7cad7c99461"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "29614ddd8c1913f303f9f139761cdcfb"
server_key_iv: "cff144bda93391b919e691a07c83c37a"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.232878 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:40:15.235337 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:15.236902 23092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:40:15.237046 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal
uuid: "5e2e01d0da7c495d86faf7cad7c99461"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "29614ddd8c1913f303f9f139761cdcfb"
server_key_iv: "cff144bda93391b919e691a07c83c37a"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.237160 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:15.246433 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:15.249665 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:15.249876 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:15.257409 23092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39523
I20260620 04:40:15.257699 23159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39523 every 8 connection(s)
I20260620 04:40:15.258803 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:40:15.261830 23160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:15.265502 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.267455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 23092
I20260620 04:40:15.267588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-1/wal/instance
I20260620 04:40:15.267892 392 external_mini_cluster.cc:1428] Setting key 034b67f7a63339d929d3db135c36f6d1
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 04:40:15.279675 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } attempt: 3
I20260620 04:40:15.280683 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.267598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.0.98.62:39555)
0620 04:40:15.267731 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.267735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.268757 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 04:40:15.269701 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.269705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.269765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.269878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.271035 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.271564 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.272796 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.273038 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.273158 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.275681 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.275690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.275693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.275718 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.278007 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.278575 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.278579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.278580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.278710 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.279128 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.279131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.279133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.279338 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.279460 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.280128 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.280537 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:40:15.281457 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.268000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39555 (local address 127.0.0.1:36154)
0620 04:40:15.268607 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.268649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.269499 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.270052 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.270058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.270405 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.270878 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.270905 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.271702 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.271710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.272666 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.272676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.273201 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.273205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.273371 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.274007 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.274024 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.275464 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.278150 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.278157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.278175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.278464 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.278819 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.278823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.278825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.279014 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.279458 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.279467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.279821 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.280064 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.280459 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":194,"threads_started":1}
I20260620 04:40:15.283001 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:40:15.293133 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.283542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47660)
0620 04:40:15.283652 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.283667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.283748 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.284234 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.284240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.284435 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.284578 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.284589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.285546 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.285553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.286063 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.286070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.286548 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.286552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.286634 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.287091 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.287104 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.288382 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.291021 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.291027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.291301 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.291754 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.291759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.291873 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.292581 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.292584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.292705 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.292818 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.292957 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 04:40:15.293133 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.280150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:47656)
0620 04:40:15.280306 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.280320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.280394 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.281481 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.281484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.281652 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.281757 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.281764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.283022 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.283029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.284009 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.284023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.284794 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.284801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.284914 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.285448 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.285469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.287836 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.291016 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.291022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.291301 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.291754 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.291759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.291862 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.292581 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.292584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.292700 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.292822 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.292957 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 04:40:15.293216 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.283685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.0.98.61:39523)
0620 04:40:15.283976 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.283982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.283992 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:40:15.284016 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.284018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.284048 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.284187 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.284716 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.285427 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.286180 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.286435 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.286550 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.288533 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.288539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.288543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.288568 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.290897 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291458 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.291465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.291466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.291602 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.292002 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.292006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.292008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.292430 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.292583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.292930 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.293056 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":6272,"thread_start_us":126,"threads_started":1}
I20260620 04:40:15.293655 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.280480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.98.61:39523)
0620 04:40:15.281062 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.281072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.281091 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:40:15.281126 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.281130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.281323 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.281432 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.281985 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.282892 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.284132 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.284615 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.284783 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.287986 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.287999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.288017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.288089 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.290865 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.291458 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.291470 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.291471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.291592 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.292002 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.292006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.292008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.292430 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.292583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.293354 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.293471 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":28928,"thread_start_us":148,"threads_started":1}
I20260620 04:40:15.296336 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.296330 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.297542 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.297280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40472)
0620 04:40:15.297428 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.297486 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 04:40:15.297636 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.297314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40474)
0620 04:40:15.297471 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.297549 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260620 04:40:15.297854 23108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
I20260620 04:40:15.298247 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.298008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40482)
0620 04:40:15.298146 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.298194 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 04:40:15.298394 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
I20260620 04:40:15.299350 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.299087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40498)
0620 04:40:15.299243 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.299295 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 04:40:15.299582 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:40:15.300609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGAOikI/build/debug/bin/kudu
/tmp/dist-test-taskGAOikI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46741
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:40:15.329571 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 1
I20260620 04:40:15.330518 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.330203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40508)
0620 04:40:15.330385 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.330444 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 04:40:15.331275 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.331053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40510)
0620 04:40:15.331174 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.331224 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 04:40:15.331415 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
I20260620 04:40:15.348861 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 1
I20260620 04:40:15.349741 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.349402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40514)
0620 04:40:15.349605 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.349666 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 04:40:15.350561 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.350326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40526)
0620 04:40:15.350459 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.350511 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 04:40:15.350771 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
I20260620 04:40:15.406592 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 2
I20260620 04:40:15.407361 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.407089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40532)
0620 04:40:15.407228 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.407291 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 04:40:15.408284 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.407960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40534)
0620 04:40:15.408108 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.408159 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 04:40:15.408447 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
W20260620 04:40:15.414410 23168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:40:15.414650 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:40:15.414711 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:40:15.414783 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:40:15.418784 23168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:40:15.418835 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:40:15.418854 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:40:15.418870 23168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:40:15.418924 23168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:40:15.418969 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 2
I20260620 04:40:15.419787 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.419427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40550)
0620 04:40:15.419665 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.419718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 04:40:15.420531 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.420345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40558)
0620 04:40:15.420443 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.420485 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 04:40:15.420727 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46741: connect: Connection refused (error 111)
I20260620 04:40:15.424592 23168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39555,127.0.98.61:39523,127.0.98.60:46741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:40:15.425585 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:40:15.426580 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:40:15.435161 23174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:15.435227 23173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:40:15.435186 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:40:15.435493 23168 server_base.cc:1034] running on GCE node
I20260620 04:40:15.435988 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:40:15.437098 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:40:15.438308 23168 hybrid_clock.cc:648] HybridClock initialized: now 1781930415438287 us; error 30 us; skew 500 ppm
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:40:15.441377 23168 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:40:15.442248 23168 webserver.cc:460] Webserver started at http://127.0.98.60:38629/ using document root <none> and password file <none>
I20260620 04:40:15.442792 23168 fs_manager.cc:359] Metadata directory not provided
I20260620 04:40:15.442888 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:40:15.443107 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:40:15.445137 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data/instance:
uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "59fd2888d14b09de3bcd64987842fa9a"
server_key_iv: "eccb2b1e4851a18ae713955ea71a92bd"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.445688 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal/instance:
uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "59fd2888d14b09de3bcd64987842fa9a"
server_key_iv: "eccb2b1e4851a18ae713955ea71a92bd"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.449337 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:40:15.451689 23183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:40:15.452970 23168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:40:15.453115 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal
uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b"
format_stamp: "Formatted at 2026-06-20 04:40:15 on dist-test-slave-zmf4"
server_key: "59fd2888d14b09de3bcd64987842fa9a"
server_key_iv: "eccb2b1e4851a18ae713955ea71a92bd"
server_key_version: "encryptionkey@0"
I20260620 04:40:15.453240 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:40:15.465188 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:40:15.468801 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:40:15.469015 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:40:15.475803 23168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46741
I20260620 04:40:15.476131 23235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46741 every 8 connection(s)
I20260620 04:40:15.476835 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:40:15.477319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGAOikI/build/debug/bin/kudu as pid 23168
I20260620 04:40:15.477492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGAOikI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781930128116734-392-0/minicluster-data/master-2/wal/instance
I20260620 04:40:15.477815 392 external_mini_cluster.cc:1428] Setting key 73d702a2fb6123f411e74eb25268d0b0
I20260620 04:40:15.480533 23236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:40:15.485380 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:40:15.493305 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.479277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.60:46741)
0620 04:40:15.479707 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.479721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.479756 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:40:15.480573 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.480590 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.480949 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.481212 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.482139 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.483632 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.484789 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.485265 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.485386 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.488412 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.488420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.488437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.488465 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.490161 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.490640 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.490645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.490650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.490803 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.491401 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.491403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.491405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.491743 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.491958 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.492291 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.492553 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":164,"threads_started":1}
I20260620 04:40:15.496896 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.487538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.98.62:39555)
0620 04:40:15.487687 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.487692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.487992 (+ 300us) server_negotiation.cc:407] Connection header received
0620 04:40:15.488135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.488138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.488217 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.488346 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.488966 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.489448 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.490644 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.490984 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.491093 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.493309 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.493312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.493314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.493335 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.494776 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.495380 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.495384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.495389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.495479 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.495791 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.495794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.495796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.496030 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.496120 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.496560 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.496685 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:40:15.496927 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.487459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39555 (local address 127.0.0.1:44466)
0620 04:40:15.487857 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.487907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.488036 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.488393 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.488397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.488717 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.488842 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.488848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.489559 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.489564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.490160 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.490168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.491093 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.491095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.491187 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.491753 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.491764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.493154 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.494944 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.494951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.494956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.495266 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.495592 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.495595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.495597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.495690 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.496151 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.496157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.496331 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.496505 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.496679 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":143,"threads_started":1}
I20260620 04:40:15.498550 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:40:15.505817 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.495454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.0.98.62:39555)
0620 04:40:15.495803 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.495812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.495823 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:40:15.495855 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.495857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.495896 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.495985 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.496837 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.497818 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.499102 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.499440 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.499567 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.502400 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.502405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.502407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.502432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.503919 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.504403 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.504406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.504408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.504484 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.504790 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.504795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.504797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.505023 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.505287 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.505501 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.505623 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260620 04:40:15.507655 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 3
I20260620 04:40:15.507858 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.499075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39523 (local address 127.0.0.1:60706)
0620 04:40:15.499181 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.499196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.499265 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.499609 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.499612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.499787 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.499921 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.499938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.500919 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.500924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.501628 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.501637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.502237 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.502242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.502356 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.502781 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.502800 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.504139 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.506162 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.506165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.506167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.506382 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.506722 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.506725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.506727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.506823 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.507363 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.507366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.507466 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.507573 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.507705 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 04:40:15.508415 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.499244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.98.61:39523)
0620 04:40:15.499358 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.499362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.499376 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:40:15.499420 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.499422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.499474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.499558 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.500184 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.500795 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.501755 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.502107 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.502220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.504292 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.504295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.504299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.504329 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.506037 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.506492 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.506497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.506499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.506614 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.506973 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.506979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.506981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.507203 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.507298 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.507659 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.507780 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:40:15.509765 23236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:40:15.512138 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } attempt: 3
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:40:15 dist-test-slave-zmf4 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:40:15.519593 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.507210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.98.61:39523)
0620 04:40:15.507321 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.507325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.507338 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:40:15.507437 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.507440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.507483 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.507567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.508354 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.509133 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.510443 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.510799 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.510885 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.514942 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.514949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.514954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.514982 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.517004 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.517659 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.517665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.517669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.517755 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.518087 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.518092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.518094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.518296 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.518493 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.519081 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.519204 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:40:15.519817 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.508034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40580)
0620 04:40:15.508408 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.508427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.508502 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.509044 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.509047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.509235 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.509337 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.509343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.510454 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.510457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.511218 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.511231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.511885 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.511889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.511975 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.512470 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.512484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.514264 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.516851 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.516857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.516859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.517657 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.518256 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.518259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.518261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.518348 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.519019 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.519023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.519124 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.519212 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.519652 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260620 04:40:15.520473 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.510212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40588)
0620 04:40:15.510293 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.510306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.510372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.510803 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.510806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.510962 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.511017 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.511021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.512424 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.512428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.513144 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.513153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.513656 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.513659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.513753 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.514479 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.514495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.515825 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.518754 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.518757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.518758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.519084 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.519391 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.519395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.519400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.519487 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.519916 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.519919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.520013 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.520116 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.520281 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":5}
I20260620 04:40:15.520994 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.510341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.98.60:46741)
0620 04:40:15.510628 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.510634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.510644 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:40:15.510668 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.510669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.510697 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.510796 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.511121 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.512276 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.513268 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.513553 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.513659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.515942 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.515946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.515951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.515978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.518657 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.519198 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.519203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.519211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.519296 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.519583 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.519586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.519587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.519773 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.519862 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.520676 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.520788 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":269,"server-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260620 04:40:15.521454 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.508719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.60:46741)
0620 04:40:15.508845 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.508848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.508859 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:40:15.508885 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.508887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.508917 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.508977 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.509468 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.510304 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.511373 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.511734 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.511847 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.514405 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.514409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.514411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.514435 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.516690 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.518041 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.518045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.518047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.518136 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.518532 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.518538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.518542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.518797 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.520913 (+ 2116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.520985 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.521301 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":55}
I20260620 04:40:15.523932 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:40:15.524569 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: Bootstrap starting.
I20260620 04:40:15.524569 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: Bootstrap starting.
I20260620 04:40:15.525481 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.512618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46741 (local address 127.0.0.1:40592)
0620 04:40:15.512841 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 04:40:15.512855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:40:15.512930 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:40:15.514062 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:40:15.514065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:40:15.514232 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:40:15.514300 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.514305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.515373 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.515377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.516136 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:40:15.516144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.516644 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.516647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:40:15.516746 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.517160 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:40:15.517176 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:40:15.518651 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:40:15.521235 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.521239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.521241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.521509 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.524324 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:40:15.524328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:40:15.524330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:40:15.524434 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:40:15.524996 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:40:15.524999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:40:15.525094 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:40:15.525191 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:40:15.525318 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 04:40:15.526273 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.513085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.0.98.60:46741)
0620 04:40:15.513786 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.513794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.513808 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:40:15.513840 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.513843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.513917 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.514044 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.514420 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.515240 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.516308 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.516534 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.516643 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.519002 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.519010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.519016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.519042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.521103 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.522132 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.522134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.522136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.522214 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.524563 (+ 2349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.524566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.524569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.524878 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.525913 (+ 1035us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.526008 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.526118 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"spinlock_wait_cycles":4480,"thread_start_us":122,"threads_started":1}
I20260620 04:40:15.525161 23059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480
I20260620 04:40:15.527280 23236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:15.527280 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:15.528146 23085 log.cc:824] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:15.528146 23236 log.cc:824] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:15.529209 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: Bootstrap starting.
I20260620 04:40:15.530442 23236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: No bootstrap required, opened a new log
I20260620 04:40:15.530442 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: No bootstrap required, opened a new log
I20260620 04:40:15.531764 23160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: Neither blocks nor log segments found. Creating new log.
I20260620 04:40:15.532676 23160 log.cc:824] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: Log is configured to *not* fsync() on all Append() calls
I20260620 04:40:15.533473 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.533708 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cbda1df9b94a02bdd89a3ebe4ef848, State: Initialized, Role: FOLLOWER
I20260620 04:40:15.534114 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.534674 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: No bootstrap required, opened a new log
I20260620 04:40:15.535001 23236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.535216 23236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9b9204f9ed4eb395765f8b6d1bed0b, State: Initialized, Role: FOLLOWER
I20260620 04:40:15.535436 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:15.535511 23059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:40:15.535523 23236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.535616 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:15.536796 23236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:15.537348 23059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.538805 23059 leader_election.cc:288] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [CANDIDATE]: Term 1 election: Requested vote from peers 5e2e01d0da7c495d86faf7cad7c99461 (127.0.98.61:39523), 0f9b9204f9ed4eb395765f8b6d1bed0b (127.0.98.60:46741)
I20260620 04:40:15.538139 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.538796 23209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b"
I20260620 04:40:15.538998 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.539033 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:15.540409 23209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cbda1df9b94a02bdd89a3ebe4ef848 in term 1.
I20260620 04:40:15.541115 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.541671 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.540978 23033 leader_election.cc:302] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9b9204f9ed4eb395765f8b6d1bed0b, 40cbda1df9b94a02bdd89a3ebe4ef848; no voters:
I20260620 04:40:15.543131 23247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:40:15.543394 23247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [term 1 LEADER]: Becoming Leader. State: Replica: 40cbda1df9b94a02bdd89a3ebe4ef848, State: Running, Role: LEADER
I20260620 04:40:15.543785 23247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.545068 23160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.545292 23160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2e01d0da7c495d86faf7cad7c99461, State: Initialized, Role: FOLLOWER
I20260620 04:40:15.545603 23160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } }
I20260620 04:40:15.545866 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40cbda1df9b94a02bdd89a3ebe4ef848. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
W20260620 04:40:15.546041 23270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:15.546146 23270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 04:40:15.546092 23269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 04:40:15.546044 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: This master's current role is: LEADER
W20260620 04:40:15.546182 23269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:15.546679 23281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:40:15.548040 23281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:40:15.548137 23160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:40:15.548220 23134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e2e01d0da7c495d86faf7cad7c99461"
I20260620 04:40:15.548439 23134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:40:15.549831 23134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cbda1df9b94a02bdd89a3ebe4ef848 in term 1.
I20260620 04:40:15.548979 23168 master_runner.cc:186] Error getting master registration for 127.0.98.61:39523: OK, instance_id { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" instance_seqno: 1781930415254847 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:40:15.552692 23168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:40:15.555553 23209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [term 1 FOLLOWER]: Refusing update from remote peer 40cbda1df9b94a02bdd89a3ebe4ef848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:40:15.556262 23247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:40:15.552837 23017 master_runner.cc:186] Error getting master registration for 127.0.98.61:39523: OK, instance_id { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" instance_seqno: 1781930415254847 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:40:15.557125 23017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:40:15.558522 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.558717 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.559988 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40cbda1df9b94a02bdd89a3ebe4ef848. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.560103 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.561235 23134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [term 1 FOLLOWER]: Refusing update from remote peer 40cbda1df9b94a02bdd89a3ebe4ef848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:40:15.561875 23247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:40:15.563143 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.542833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.0.98.62:39555)
0620 04:40:15.542940 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.542946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.550837 (+ 7891us) server_negotiation.cc:407] Connection header received
0620 04:40:15.551015 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.551018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.551064 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.551112 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.552120 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.552904 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.554072 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.554423 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.554495 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.558922 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.558925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.558928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.558955 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.561201 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.561751 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.561754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.561756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.561849 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.562202 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.562206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.562208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.562427 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.562537 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.562872 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.562992 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:40:15.565778 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40cbda1df9b94a02bdd89a3ebe4ef848. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.565891 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 04:40:15.567744 23293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:40:15.567807 23293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:40:15.576350 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.543765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.98.60:46741)
0620 04:40:15.543870 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.543874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.555194 (+ 11320us) server_negotiation.cc:407] Connection header received
0620 04:40:15.555376 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.555379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.555442 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.555493 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.556450 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.557269 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.558488 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.558849 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.558923 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.572454 (+ 13531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.572458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.572460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.572493 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.574158 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.574943 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.574948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.574952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.575052 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.575358 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.575361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.575362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.575578 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.575704 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.576038 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.576163 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:40:15.580175 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:15.550412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.61:39523)
0620 04:40:15.550550 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.550555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.559582 (+ 9027us) server_negotiation.cc:407] Connection header received
0620 04:40:15.559793 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.559798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.559860 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.559911 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.561013 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.561769 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.562858 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:15.563197 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:15.563286 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:15.569524 (+ 6238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:40:15.569529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:40:15.569534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:40:15.569561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:40:15.571440 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.578748 (+ 7308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.578767 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.578769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.578868 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:40:15.579172 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:40:15.579175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:40:15.579176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:40:15.579413 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:40:15.579566 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:15.579908 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 04:40:15.580034 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 04:40:15.581801 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.581943 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:15.582111 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.582223 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.584793 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.584909 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848 [sys.catalog]: This master's current role is: LEADER
I20260620 04:40:15.585988 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40cbda1df9b94a02bdd89a3ebe4ef848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39555 } } peers { permanent_uuid: "5e2e01d0da7c495d86faf7cad7c99461" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39523 } } peers { permanent_uuid: "0f9b9204f9ed4eb395765f8b6d1bed0b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46741 } } }
I20260620 04:40:15.586102 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:40:15.586330 23298 mvcc.cc:205] Tried to move back new op lower bound from 7298786982109294592 to 7298786982071377920. Current Snapshot: MvccSnapshot[applied={T|T < 7298786982109294592}]
I20260620 04:40:15.588328 23281 catalog_manager.cc:1347] Generated new cluster ID: e469b4d6c10143419f3973884161395a
I20260620 04:40:15.588402 23281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:40:15.603694 23281 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:40:15.606455 23281 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:40:15.623209 23281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40cbda1df9b94a02bdd89a3ebe4ef848: Generated new TSK 0
I20260620 04:40:15.624063 23281 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:40:15.660211 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.658362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.98.61:39523)
0620 04:40:15.658505 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.658510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.659114 (+ 604us) server_negotiation.cc:407] Connection header received
0620 04:40:15.659306 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.659309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.659358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.659409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.660044 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60732: BlockingRecv error: recv got EOF from 127.0.0.1:60732 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 04:40:15.662604 23241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.660984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.0.98.62:39555)
0620 04:40:15.661095 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.661099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.661725 (+ 626us) server_negotiation.cc:407] Connection header received
0620 04:40:15.661892 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.661895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.661941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.661991 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.662487 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44502: BlockingRecv error: recv got EOF from 127.0.0.1:44502 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:40:15.664014 23243 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.661474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.0.98.60:46741)
0620 04:40:15.661580 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.661584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.663176 (+ 1592us) server_negotiation.cc:407] Connection header received
0620 04:40:15.663333 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.663336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.663393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.663442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.663903 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40614: BlockingRecv error: recv got EOF from 127.0.0.1:40614 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:40:15.668092 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.666547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.0.98.61:39523)
0620 04:40:15.666645 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.666649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.667150 (+ 501us) server_negotiation.cc:407] Connection header received
0620 04:40:15.667371 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.667375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.667452 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.667505 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.667995 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60748: BlockingRecv error: recv got EOF from 127.0.0.1:60748 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 04:40:15.672824 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.671301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.0.98.61:39523)
0620 04:40:15.671414 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.671418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.671920 (+ 502us) server_negotiation.cc:407] Connection header received
0620 04:40:15.672093 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.672097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.672143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.672218 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.672729 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60754: BlockingRecv error: recv got EOF from 127.0.0.1:60754 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 04:40:15.678122 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.676660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.98.61:39523)
0620 04:40:15.676778 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.676782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.677275 (+ 493us) server_negotiation.cc:407] Connection header received
0620 04:40:15.677467 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.677470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.677517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.677564 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.678026 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60762: BlockingRecv error: recv got EOF from 127.0.0.1:60762 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 04:40:15.685554 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.683975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.0.98.61:39523)
0620 04:40:15.684104 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.684108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.684696 (+ 588us) server_negotiation.cc:407] Connection header received
0620 04:40:15.684871 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.684874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.684921 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.684970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.685454 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60766: BlockingRecv error: recv got EOF from 127.0.0.1:60766 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 04:40:15.697506 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.695544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.98.61:39523)
0620 04:40:15.695648 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.695653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.696178 (+ 525us) server_negotiation.cc:407] Connection header received
0620 04:40:15.696402 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.696406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.696455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.696503 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.697401 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60768: BlockingRecv error: recv got EOF from 127.0.0.1:60768 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:40:15.717794 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.715882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.98.61:39523)
0620 04:40:15.716010 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.716014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.716671 (+ 657us) server_negotiation.cc:407] Connection header received
0620 04:40:15.716916 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.716920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.716975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.717028 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.717677 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60782: BlockingRecv error: recv got EOF from 127.0.0.1:60782 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:40:15.754805 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.753171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.0.98.61:39523)
0620 04:40:15.753287 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.753292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.753850 (+ 558us) server_negotiation.cc:407] Connection header received
0620 04:40:15.754045 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.754049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.754101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.754151 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.754690 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60788: BlockingRecv error: recv got EOF from 127.0.0.1:60788 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:40:15.822474 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.821270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.0.98.61:39523)
0620 04:40:15.821413 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.821417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.821599 (+ 182us) server_negotiation.cc:407] Connection header received
0620 04:40:15.821717 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.821720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.821767 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.821849 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.822318 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60804: BlockingRecv error: recv got EOF from 127.0.0.1:60804 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 04:40:15.953788 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:15.952597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.0.98.61:39523)
0620 04:40:15.952744 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:15.952748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:15.952886 (+ 138us) server_negotiation.cc:407] Connection header received
0620 04:40:15.952992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:15.952995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:15.953041 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:15.953115 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:15.953671 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60812: BlockingRecv error: recv got EOF from 127.0.0.1:60812 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 04:40:16.213408 23166 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:40:16.212151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.98.61:39523)
0620 04:40:16.212317 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.212322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.212431 (+ 109us) server_negotiation.cc:407] Connection header received
0620 04:40:16.212615 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.212621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.212674 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.212809 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:40:16.213268 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60826: BlockingRecv error: recv got EOF from 127.0.0.1:60826 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 04:40:16.547514 23269 catalog_manager.cc:1259] Loaded cluster ID: e469b4d6c10143419f3973884161395a
I20260620 04:40:16.547598 23269 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: loading cluster ID for follower catalog manager: success
I20260620 04:40:16.549434 23269 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: acquiring CA information for follower catalog manager: success
I20260620 04:40:16.550596 23269 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f9b9204f9ed4eb395765f8b6d1bed0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:40:16.569077 23293 catalog_manager.cc:1259] Loaded cluster ID: e469b4d6c10143419f3973884161395a
I20260620 04:40:16.569152 23293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: loading cluster ID for follower catalog manager: success
I20260620 04:40:16.570735 23293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: acquiring CA information for follower catalog manager: success
I20260620 04:40:16.571841 23293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e2e01d0da7c495d86faf7cad7c99461: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:40:16.731848 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:16.727725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.98.61:39523)
0620 04:40:16.728000 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.728004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.728019 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:40:16.728070 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.728073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.728251 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.728365 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:16.728702 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.729311 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.730211 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.730453 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.730543 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:16.730910 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:16.731367 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:16.731470 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:16.731657 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 04:40:16.731711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260620 04:40:16.740280 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:16.735112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.98.60:46741)
0620 04:40:16.735423 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.735426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.735440 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:40:16.735518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.735520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.735626 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.735722 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:16.736159 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.736983 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.738555 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.738965 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.739058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:16.739377 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:16.739817 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:16.739978 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:16.740022 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 04:40:16.740080 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260620 04:40:16.747318 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:16.743666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.98.61:39523)
0620 04:40:16.743784 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.743788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.743802 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:40:16.743922 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.743927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.743985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.744070 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:16.744353 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.745091 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.746106 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.746391 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.746503 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:16.746758 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:16.746833 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:16.746926 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:16.747087 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:40:16.747148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:40:16.748689 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:16.744517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.98.60:46741)
0620 04:40:16.744642 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.744645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.744907 (+ 262us) server_negotiation.cc:407] Connection header received
0620 04:40:16.745071 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.745074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.745124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.745211 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:16.745530 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.746169 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.747454 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.747786 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.747910 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:16.748178 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:16.748271 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:16.748316 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:16.748477 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:40:16.748551 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:40:16.748975 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:40:16.743770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.98.62:39555)
0620 04:40:16.744063 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 04:40:16.744068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:40:16.744323 (+ 255us) server_negotiation.cc:407] Connection header received
0620 04:40:16.744563 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:40:16.744567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:40:16.744720 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:40:16.744840 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:40:16.745089 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.745774 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.746919 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:40:16.747258 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:40:16.747381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:40:16.748080 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:40:16.748459 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:40:16.748540 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:40:16.748807 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 04:40:16.748869 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260620 04:40:16.752142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 23017
I20260620 04:40:16.759102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 23092
I20260620 04:40:16.765254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGAOikI/build/debug/bin/kudu with pid 23168
2026-06-20T04:40:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3124 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6329 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288641 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST